2024-04-12T21:41:41.425 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-12T21:41:41.425 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_teuthology@teuthology --name teuthology-2024-04-12_21:08:02-orch-squid-distro-default-smithi --archive /home/teuthworker/archive/teuthology-2024-04-12_21:08:02-orch-squid-distro-default-smithi/7654552 --description orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} --verbose -- /home/teuthworker/archive/teuthology-2024-04-12_21:08:02-orch-squid-distro-default-smithi/7654552/orig.config.yaml 2024-04-12T21:41:41.456 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-12T21:41:41.585 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-12_21:08:02-orch-squid-distro-default-smithi/7654552 branch: squid description: orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} email: ceph-qa@ceph.com first_in_suite: false job_id: '7654552' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-04-12_21:08:02-orch-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: false mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: b277feb2c7beb4597d8ecc39d58d5c36b81f613a ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: b277feb2c7beb4597d8ecc39d58d5c36b81f613a workunit: branch: squid sha1: 98423fadeecbba811088c3136f8295dae6dfe870 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b seed: 2723 sha1: b277feb2c7beb4597d8ecc39d58d5c36b81f613a sleep_before_teardown: 0 subset: 24/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_98423fadeecbba811088c3136f8295dae6dfe870/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 98423fadeecbba811088c3136f8295dae6dfe870 targets: smithi097.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG4eVILE1reKOrx59u+2caNLf58FHQ/N1J8NVOvd7RU2mNjNDFKyMY0rXLvNNV3W9KpF2Xv3F3BqBiJmBTJiWhU= smithi114.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEGezFQLvdH5Ts9lZM2TglSlf9zAepCCLzbYdhsV+9c8NMLFhO74WYfXhVBhKrMFB3dn6rA3xKU8AZIysFjNlOQ= tasks: - cephadm: cephadm_branch: v17.2.0 cephadm_git_url: https://github.com/ceph/ceph image: quay.io/ceph/ceph:v17.2.0 - cephadm.shell: mon.a: - ceph config set mgr mgr/cephadm/use_repo_digest false --force - cephadm.shell: env: - sha1 mon.a: - radosgw-admin realm create --rgw-realm=r --default - radosgw-admin zonegroup create --rgw-zonegroup=default --master --default - radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default - radosgw-admin period update --rgw-realm=r --commit - ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000 - ceph orch apply rgw smpl - ceph osd pool create foo - rbd pool init foo - ceph orch apply iscsi foo u p - sleep 120 - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 mon.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph orch upgrade status - ceph health detail - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - ceph orch ls | grep '^osd ' - cephadm.shell: mon.a: - ceph orch upgrade ls - ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0 - ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-12_21:08:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-12T21:41:41.585 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_98423fadeecbba811088c3136f8295dae6dfe870/qa; will attempt to use it 2024-04-12T21:41:41.586 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_98423fadeecbba811088c3136f8295dae6dfe870/qa/tasks 2024-04-12T21:41:41.586 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-12T21:41:41.587 INFO:teuthology.task.internal:Checking packages... 2024-04-12T21:41:41.612 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'b277feb2c7beb4597d8ecc39d58d5c36b81f613a' 2024-04-12T21:41:41.612 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-12T21:41:41.612 INFO:teuthology.packaging:ref: None 2024-04-12T21:41:41.612 INFO:teuthology.packaging:tag: None 2024-04-12T21:41:41.612 INFO:teuthology.packaging:branch: squid 2024-04-12T21:41:41.612 INFO:teuthology.packaging:sha1: b277feb2c7beb4597d8ecc39d58d5c36b81f613a 2024-04-12T21:41:41.613 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-04-12T21:41:41.919 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2053.gb277feb2 2024-04-12T21:41:41.921 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-12T21:41:41.927 INFO:teuthology.task.internal:no buildpackages task found 2024-04-12T21:41:41.927 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-12T21:41:42.024 INFO:teuthology.task.internal:Saving configuration 2024-04-12T21:41:42.036 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-12T21:41:42.042 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-12T21:41:42.067 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi097.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-12_21:08:02-orch-squid-distro-default-smithi/7654552', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-12 21:36:33.259804', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG4eVILE1reKOrx59u+2caNLf58FHQ/N1J8NVOvd7RU2mNjNDFKyMY0rXLvNNV3W9KpF2Xv3F3BqBiJmBTJiWhU='} 2024-04-12T21:41:42.106 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi114.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-12_21:08:02-orch-squid-distro-default-smithi/7654552', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-12 21:36:33.258584', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEGezFQLvdH5Ts9lZM2TglSlf9zAepCCLzbYdhsV+9c8NMLFhO74WYfXhVBhKrMFB3dn6rA3xKU8AZIysFjNlOQ='} 2024-04-12T21:41:42.106 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-12T21:41:42.119 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2024-04-12T21:41:42.119 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-04-12T21:41:42.119 INFO:teuthology.run_tasks:Running task console_log... 2024-04-12T21:41:42.217 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6bbca600d0>, signals=[15]) 2024-04-12T21:41:42.218 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-12T21:41:42.226 INFO:teuthology.task.internal:Opening connections... 2024-04-12T21:41:42.226 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi097.front.sepia.ceph.com 2024-04-12T21:41:42.228 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-12T21:41:42.316 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi114.front.sepia.ceph.com 2024-04-12T21:41:42.317 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-12T21:41:42.402 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-12T21:41:42.409 DEBUG:teuthology.orchestra.run.smithi097:> uname -m 2024-04-12T21:41:42.426 INFO:teuthology.orchestra.run.smithi097.stdout:x86_64 2024-04-12T21:41:42.426 DEBUG:teuthology.orchestra.run.smithi097:> cat /etc/os-release 2024-04-12T21:41:42.482 INFO:teuthology.orchestra.run.smithi097.stdout:NAME="CentOS Stream" 2024-04-12T21:41:42.482 INFO:teuthology.orchestra.run.smithi097.stdout:VERSION="9" 2024-04-12T21:41:42.482 INFO:teuthology.orchestra.run.smithi097.stdout:ID="centos" 2024-04-12T21:41:42.482 INFO:teuthology.orchestra.run.smithi097.stdout:ID_LIKE="rhel fedora" 2024-04-12T21:41:42.483 INFO:teuthology.orchestra.run.smithi097.stdout:VERSION_ID="9" 2024-04-12T21:41:42.483 INFO:teuthology.orchestra.run.smithi097.stdout:PLATFORM_ID="platform:el9" 2024-04-12T21:41:42.483 INFO:teuthology.orchestra.run.smithi097.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-12T21:41:42.483 INFO:teuthology.orchestra.run.smithi097.stdout:ANSI_COLOR="0;31" 2024-04-12T21:41:42.483 INFO:teuthology.orchestra.run.smithi097.stdout:LOGO="fedora-logo-icon" 2024-04-12T21:41:42.483 INFO:teuthology.orchestra.run.smithi097.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-12T21:41:42.483 INFO:teuthology.orchestra.run.smithi097.stdout:HOME_URL="https://centos.org/" 2024-04-12T21:41:42.483 INFO:teuthology.orchestra.run.smithi097.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-12T21:41:42.483 INFO:teuthology.orchestra.run.smithi097.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-12T21:41:42.483 INFO:teuthology.orchestra.run.smithi097.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-12T21:41:42.484 INFO:teuthology.lock.ops:Updating smithi097.front.sepia.ceph.com on lock server 2024-04-12T21:41:42.506 DEBUG:teuthology.orchestra.run.smithi114:> uname -m 2024-04-12T21:41:42.523 INFO:teuthology.orchestra.run.smithi114.stdout:x86_64 2024-04-12T21:41:42.523 DEBUG:teuthology.orchestra.run.smithi114:> cat /etc/os-release 2024-04-12T21:41:42.578 INFO:teuthology.orchestra.run.smithi114.stdout:NAME="CentOS Stream" 2024-04-12T21:41:42.578 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION="9" 2024-04-12T21:41:42.578 INFO:teuthology.orchestra.run.smithi114.stdout:ID="centos" 2024-04-12T21:41:42.578 INFO:teuthology.orchestra.run.smithi114.stdout:ID_LIKE="rhel fedora" 2024-04-12T21:41:42.578 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION_ID="9" 2024-04-12T21:41:42.578 INFO:teuthology.orchestra.run.smithi114.stdout:PLATFORM_ID="platform:el9" 2024-04-12T21:41:42.578 INFO:teuthology.orchestra.run.smithi114.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-12T21:41:42.578 INFO:teuthology.orchestra.run.smithi114.stdout:ANSI_COLOR="0;31" 2024-04-12T21:41:42.578 INFO:teuthology.orchestra.run.smithi114.stdout:LOGO="fedora-logo-icon" 2024-04-12T21:41:42.578 INFO:teuthology.orchestra.run.smithi114.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-12T21:41:42.579 INFO:teuthology.orchestra.run.smithi114.stdout:HOME_URL="https://centos.org/" 2024-04-12T21:41:42.579 INFO:teuthology.orchestra.run.smithi114.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-12T21:41:42.579 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-12T21:41:42.579 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-12T21:41:42.579 INFO:teuthology.lock.ops:Updating smithi114.front.sepia.ceph.com on lock server 2024-04-12T21:41:42.605 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-12T21:41:42.616 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-12T21:41:42.621 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-12T21:41:42.621 DEBUG:teuthology.orchestra.run.smithi097:> test '!' -e /home/ubuntu/cephtest 2024-04-12T21:41:42.623 DEBUG:teuthology.orchestra.run.smithi114:> test '!' -e /home/ubuntu/cephtest 2024-04-12T21:41:42.640 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-12T21:41:42.646 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-12T21:41:42.646 DEBUG:teuthology.orchestra.run.smithi097:> test -z $(ls -A /var/lib/ceph) 2024-04-12T21:41:42.679 DEBUG:teuthology.orchestra.run.smithi114:> test -z $(ls -A /var/lib/ceph) 2024-04-12T21:41:42.805 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-12T21:41:42.851 INFO:teuthology.run_tasks:Running task kernel... 2024-04-12T21:41:42.867 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-12T21:41:42.867 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-12T21:41:42.867 DEBUG:teuthology.orchestra.run.smithi097:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-12T21:41:42.868 DEBUG:teuthology.orchestra.run.smithi114:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-12T21:41:42.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-12T21:41:42.885 DEBUG:teuthology.orchestra.run.smithi097:> uname -r 2024-04-12T21:41:42.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-12T21:41:42.886 DEBUG:teuthology.orchestra.run.smithi114:> uname -r 2024-04-12T21:41:42.940 INFO:teuthology.orchestra.run.smithi114.stdout:5.14.0-361.el9.x86_64 2024-04-12T21:41:42.940 INFO:teuthology.task.kernel:Running kernel on smithi114: 5.14.0-361.el9.x86_64 2024-04-12T21:41:42.940 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum install -y kernel 2024-04-12T21:41:42.942 INFO:teuthology.orchestra.run.smithi097.stdout:5.14.0-361.el9.x86_64 2024-04-12T21:41:42.942 INFO:teuthology.task.kernel:Running kernel on smithi097: 5.14.0-361.el9.x86_64 2024-04-12T21:41:42.942 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum install -y kernel 2024-04-12T21:41:45.255 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS-9 - AppStream 13 MB/s | 8.1 MB 00:00 2024-04-12T21:41:45.256 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS-9 - AppStream 13 MB/s | 8.1 MB 00:00 2024-04-12T21:41:46.999 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS-9 - BaseOS 6.3 MB/s | 2.5 MB 00:00 2024-04-12T21:41:47.163 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS-9 - BaseOS 4.3 MB/s | 2.5 MB 00:00 2024-04-12T21:41:51.272 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - BaseOS 2.1 MB/s | 8.0 MB 00:03 2024-04-12T21:41:53.123 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - AppStream 26 MB/s | 19 MB 00:00 2024-04-12T21:41:56.544 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - BaseOS 922 kB/s | 8.0 MB 00:08 2024-04-12T21:41:57.990 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - CRB 4.8 MB/s | 6.2 MB 00:01 2024-04-12T21:41:58.504 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - AppStream 23 MB/s | 19 MB 00:00 2024-04-12T21:41:59.557 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 16 kB 00:00 2024-04-12T21:42:02.643 INFO:teuthology.orchestra.run.smithi097.stdout:Extra Packages for Enterprise Linux 7.1 MB/s | 21 MB 00:02 2024-04-12T21:42:03.771 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - CRB 3.8 MB/s | 6.2 MB 00:01 2024-04-12T21:42:05.189 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - Extras packages 74 kB/s | 16 kB 00:00 2024-04-12T21:42:06.157 INFO:teuthology.orchestra.run.smithi114.stdout:Extra Packages for Enterprise Linux 25 MB/s | 21 MB 00:00 2024-04-12T21:42:07.701 INFO:teuthology.orchestra.run.smithi097.stdout:lab-extras 46 kB/s | 1.7 kB 00:00 2024-04-12T21:42:09.932 INFO:teuthology.orchestra.run.smithi097.stderr:Modular dependency problems: 2024-04-12T21:42:09.933 INFO:teuthology.orchestra.run.smithi097.stderr: 2024-04-12T21:42:09.933 INFO:teuthology.orchestra.run.smithi097.stderr: Problem 1: conflicting requests 2024-04-12T21:42:09.933 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.933 INFO:teuthology.orchestra.run.smithi097.stderr: Problem 2: conflicting requests 2024-04-12T21:42:09.933 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.933 INFO:teuthology.orchestra.run.smithi097.stderr: Problem 3: conflicting requests 2024-04-12T21:42:09.933 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.933 INFO:teuthology.orchestra.run.smithi097.stderr: Problem 4: conflicting requests 2024-04-12T21:42:09.933 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.933 INFO:teuthology.orchestra.run.smithi097.stderr: Problem 5: conflicting requests 2024-04-12T21:42:09.933 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.934 INFO:teuthology.orchestra.run.smithi097.stderr: Problem 6: conflicting requests 2024-04-12T21:42:09.934 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.934 INFO:teuthology.orchestra.run.smithi097.stderr: Problem 7: conflicting requests 2024-04-12T21:42:09.934 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.934 INFO:teuthology.orchestra.run.smithi097.stderr: Problem 8: conflicting requests 2024-04-12T21:42:09.934 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.934 INFO:teuthology.orchestra.run.smithi097.stderr: Problem 9: conflicting requests 2024-04-12T21:42:09.934 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.934 INFO:teuthology.orchestra.run.smithi097.stderr: Problem 10: conflicting requests 2024-04-12T21:42:09.934 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.934 INFO:teuthology.orchestra.run.smithi097.stderr: Problem 11: conflicting requests 2024-04-12T21:42:09.934 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.935 INFO:teuthology.orchestra.run.smithi097.stderr: Problem 12: conflicting requests 2024-04-12T21:42:09.935 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.935 INFO:teuthology.orchestra.run.smithi097.stderr: Problem 13: conflicting requests 2024-04-12T21:42:09.935 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.935 INFO:teuthology.orchestra.run.smithi097.stderr: Problem 14: conflicting requests 2024-04-12T21:42:09.935 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.935 INFO:teuthology.orchestra.run.smithi097.stderr: Problem 15: conflicting requests 2024-04-12T21:42:09.935 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.935 INFO:teuthology.orchestra.run.smithi097.stderr: Problem 16: conflicting requests 2024-04-12T21:42:09.935 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.935 INFO:teuthology.orchestra.run.smithi097.stderr: Problem 17: conflicting requests 2024-04-12T21:42:09.935 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.935 INFO:teuthology.orchestra.run.smithi097.stderr: Problem 18: conflicting requests 2024-04-12T21:42:09.936 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.936 INFO:teuthology.orchestra.run.smithi097.stderr: Problem 19: conflicting requests 2024-04-12T21:42:09.936 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.936 INFO:teuthology.orchestra.run.smithi097.stderr: Problem 20: conflicting requests 2024-04-12T21:42:09.936 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.936 INFO:teuthology.orchestra.run.smithi097.stderr: Problem 21: conflicting requests 2024-04-12T21:42:09.936 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.936 INFO:teuthology.orchestra.run.smithi097.stderr: Problem 22: conflicting requests 2024-04-12T21:42:09.936 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.936 INFO:teuthology.orchestra.run.smithi097.stderr: Problem 23: conflicting requests 2024-04-12T21:42:09.936 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.936 INFO:teuthology.orchestra.run.smithi097.stderr: Problem 24: conflicting requests 2024-04-12T21:42:09.936 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.937 INFO:teuthology.orchestra.run.smithi097.stderr: Problem 25: conflicting requests 2024-04-12T21:42:09.937 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.937 INFO:teuthology.orchestra.run.smithi097.stderr: Problem 26: conflicting requests 2024-04-12T21:42:09.937 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.937 INFO:teuthology.orchestra.run.smithi097.stderr: Problem 27: conflicting requests 2024-04-12T21:42:09.937 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.937 INFO:teuthology.orchestra.run.smithi097.stderr: Problem 28: conflicting requests 2024-04-12T21:42:09.937 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.937 INFO:teuthology.orchestra.run.smithi097.stderr: Problem 29: conflicting requests 2024-04-12T21:42:09.937 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.937 INFO:teuthology.orchestra.run.smithi097.stderr: Problem 30: conflicting requests 2024-04-12T21:42:09.937 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.937 INFO:teuthology.orchestra.run.smithi097.stderr: Problem 31: conflicting requests 2024-04-12T21:42:09.937 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.938 INFO:teuthology.orchestra.run.smithi097.stderr: Problem 32: conflicting requests 2024-04-12T21:42:09.938 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.938 INFO:teuthology.orchestra.run.smithi097.stderr: Problem 33: conflicting requests 2024-04-12T21:42:09.938 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.938 INFO:teuthology.orchestra.run.smithi097.stderr: Problem 34: conflicting requests 2024-04-12T21:42:09.938 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.938 INFO:teuthology.orchestra.run.smithi097.stderr: Problem 35: conflicting requests 2024-04-12T21:42:09.938 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.938 INFO:teuthology.orchestra.run.smithi097.stderr: Problem 36: conflicting requests 2024-04-12T21:42:09.938 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.938 INFO:teuthology.orchestra.run.smithi097.stderr: Problem 37: conflicting requests 2024-04-12T21:42:09.938 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.938 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.939 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.939 INFO:teuthology.orchestra.run.smithi097.stderr: Problem 38: cannot install the best candidate for the job 2024-04-12T21:42:09.939 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.939 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.939 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.939 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.939 INFO:teuthology.orchestra.run.smithi097.stderr: Problem 39: conflicting requests 2024-04-12T21:42:09.939 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.939 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.939 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.939 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.939 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.939 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.939 INFO:teuthology.orchestra.run.smithi097.stderr: Problem 40: cannot install the best candidate for the job 2024-04-12T21:42:09.940 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.940 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.940 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.940 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.940 INFO:teuthology.orchestra.run.smithi097.stderr: Problem 41: cannot install the best candidate for the job 2024-04-12T21:42:09.940 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.940 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.940 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.940 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.940 INFO:teuthology.orchestra.run.smithi097.stderr: Problem 42: cannot install the best candidate for the job 2024-04-12T21:42:09.940 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.940 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.940 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.941 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.941 INFO:teuthology.orchestra.run.smithi097.stderr: Problem 43: cannot install the best candidate for the job 2024-04-12T21:42:09.941 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.941 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.941 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.941 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.941 INFO:teuthology.orchestra.run.smithi097.stderr: Problem 44: cannot install the best candidate for the job 2024-04-12T21:42:09.941 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.941 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.941 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.941 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.941 INFO:teuthology.orchestra.run.smithi097.stderr: Problem 45: cannot install the best candidate for the job 2024-04-12T21:42:09.941 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.942 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.942 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-12T21:42:09.942 INFO:teuthology.orchestra.run.smithi097.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-12T21:42:10.316 INFO:teuthology.orchestra.run.smithi097.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-12T21:42:10.316 INFO:teuthology.orchestra.run.smithi097.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-12T21:42:10.418 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-04-12T21:42:10.428 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-04-12T21:42:10.429 INFO:teuthology.orchestra.run.smithi097.stdout: Package Architecture Version Repository Size 2024-04-12T21:42:10.429 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-04-12T21:42:10.429 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-04-12T21:42:10.429 INFO:teuthology.orchestra.run.smithi097.stdout: kernel x86_64 5.14.0-435.el9 baseos 214 k 2024-04-12T21:42:10.429 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2024-04-12T21:42:10.429 INFO:teuthology.orchestra.run.smithi097.stdout: kernel-core x86_64 5.14.0-435.el9 baseos 15 M 2024-04-12T21:42:10.429 INFO:teuthology.orchestra.run.smithi097.stdout: kernel-modules x86_64 5.14.0-435.el9 baseos 34 M 2024-04-12T21:42:10.429 INFO:teuthology.orchestra.run.smithi097.stdout: kernel-modules-core x86_64 5.14.0-435.el9 baseos 28 M 2024-04-12T21:42:10.429 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-12T21:42:10.429 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-04-12T21:42:10.429 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-04-12T21:42:10.429 INFO:teuthology.orchestra.run.smithi097.stdout:Install 4 Packages 2024-04-12T21:42:10.429 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-12T21:42:10.430 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 77 M 2024-04-12T21:42:10.430 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 123 M 2024-04-12T21:42:10.430 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-04-12T21:42:10.944 INFO:teuthology.orchestra.run.smithi097.stdout:(1/4): kernel-5.14.0-435.el9.x86_64.rpm 897 kB/s | 214 kB 00:00 2024-04-12T21:42:11.299 INFO:teuthology.orchestra.run.smithi114.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-04-12T21:42:12.119 INFO:teuthology.orchestra.run.smithi097.stdout:(2/4): kernel-core-5.14.0-435.el9.x86_64.rpm 11 MB/s | 15 MB 00:01 2024-04-12T21:42:12.736 INFO:teuthology.orchestra.run.smithi097.stdout:(3/4): kernel-modules-core-5.14.0-435.el9.x86_6 15 MB/s | 28 MB 00:01 2024-04-12T21:42:13.045 INFO:teuthology.orchestra.run.smithi097.stdout:(4/4): kernel-modules-5.14.0-435.el9.x86_64.rpm 14 MB/s | 34 MB 00:02 2024-04-12T21:42:13.045 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-04-12T21:42:13.045 INFO:teuthology.orchestra.run.smithi097.stdout:Total 29 MB/s | 77 MB 00:02 2024-04-12T21:42:13.520 INFO:teuthology.orchestra.run.smithi114.stderr:Modular dependency problems: 2024-04-12T21:42:13.520 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-04-12T21:42:13.520 INFO:teuthology.orchestra.run.smithi114.stderr: Problem 1: conflicting requests 2024-04-12T21:42:13.520 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.521 INFO:teuthology.orchestra.run.smithi114.stderr: Problem 2: conflicting requests 2024-04-12T21:42:13.521 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.521 INFO:teuthology.orchestra.run.smithi114.stderr: Problem 3: conflicting requests 2024-04-12T21:42:13.521 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.521 INFO:teuthology.orchestra.run.smithi114.stderr: Problem 4: conflicting requests 2024-04-12T21:42:13.521 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.521 INFO:teuthology.orchestra.run.smithi114.stderr: Problem 5: conflicting requests 2024-04-12T21:42:13.521 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.521 INFO:teuthology.orchestra.run.smithi114.stderr: Problem 6: conflicting requests 2024-04-12T21:42:13.521 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.521 INFO:teuthology.orchestra.run.smithi114.stderr: Problem 7: conflicting requests 2024-04-12T21:42:13.521 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.522 INFO:teuthology.orchestra.run.smithi114.stderr: Problem 8: conflicting requests 2024-04-12T21:42:13.522 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.522 INFO:teuthology.orchestra.run.smithi114.stderr: Problem 9: conflicting requests 2024-04-12T21:42:13.522 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.522 INFO:teuthology.orchestra.run.smithi114.stderr: Problem 10: conflicting requests 2024-04-12T21:42:13.522 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.522 INFO:teuthology.orchestra.run.smithi114.stderr: Problem 11: conflicting requests 2024-04-12T21:42:13.522 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.522 INFO:teuthology.orchestra.run.smithi114.stderr: Problem 12: conflicting requests 2024-04-12T21:42:13.522 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.522 INFO:teuthology.orchestra.run.smithi114.stderr: Problem 13: conflicting requests 2024-04-12T21:42:13.522 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.522 INFO:teuthology.orchestra.run.smithi114.stderr: Problem 14: conflicting requests 2024-04-12T21:42:13.522 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.523 INFO:teuthology.orchestra.run.smithi114.stderr: Problem 15: conflicting requests 2024-04-12T21:42:13.523 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.523 INFO:teuthology.orchestra.run.smithi114.stderr: Problem 16: conflicting requests 2024-04-12T21:42:13.523 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.523 INFO:teuthology.orchestra.run.smithi114.stderr: Problem 17: conflicting requests 2024-04-12T21:42:13.523 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.523 INFO:teuthology.orchestra.run.smithi114.stderr: Problem 18: conflicting requests 2024-04-12T21:42:13.523 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.523 INFO:teuthology.orchestra.run.smithi114.stderr: Problem 19: conflicting requests 2024-04-12T21:42:13.523 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.523 INFO:teuthology.orchestra.run.smithi114.stderr: Problem 20: conflicting requests 2024-04-12T21:42:13.523 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.523 INFO:teuthology.orchestra.run.smithi114.stderr: Problem 21: conflicting requests 2024-04-12T21:42:13.523 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.523 INFO:teuthology.orchestra.run.smithi114.stderr: Problem 22: conflicting requests 2024-04-12T21:42:13.524 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.524 INFO:teuthology.orchestra.run.smithi114.stderr: Problem 23: conflicting requests 2024-04-12T21:42:13.524 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.524 INFO:teuthology.orchestra.run.smithi114.stderr: Problem 24: conflicting requests 2024-04-12T21:42:13.524 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.524 INFO:teuthology.orchestra.run.smithi114.stderr: Problem 25: conflicting requests 2024-04-12T21:42:13.524 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.524 INFO:teuthology.orchestra.run.smithi114.stderr: Problem 26: conflicting requests 2024-04-12T21:42:13.524 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.524 INFO:teuthology.orchestra.run.smithi114.stderr: Problem 27: conflicting requests 2024-04-12T21:42:13.524 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.524 INFO:teuthology.orchestra.run.smithi114.stderr: Problem 28: conflicting requests 2024-04-12T21:42:13.524 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.524 INFO:teuthology.orchestra.run.smithi114.stderr: Problem 29: conflicting requests 2024-04-12T21:42:13.525 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.525 INFO:teuthology.orchestra.run.smithi114.stderr: Problem 30: conflicting requests 2024-04-12T21:42:13.525 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.525 INFO:teuthology.orchestra.run.smithi114.stderr: Problem 31: conflicting requests 2024-04-12T21:42:13.525 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.525 INFO:teuthology.orchestra.run.smithi114.stderr: Problem 32: conflicting requests 2024-04-12T21:42:13.525 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.525 INFO:teuthology.orchestra.run.smithi114.stderr: Problem 33: conflicting requests 2024-04-12T21:42:13.525 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.525 INFO:teuthology.orchestra.run.smithi114.stderr: Problem 34: conflicting requests 2024-04-12T21:42:13.525 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.525 INFO:teuthology.orchestra.run.smithi114.stderr: Problem 35: conflicting requests 2024-04-12T21:42:13.525 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.525 INFO:teuthology.orchestra.run.smithi114.stderr: Problem 36: conflicting requests 2024-04-12T21:42:13.526 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.526 INFO:teuthology.orchestra.run.smithi114.stderr: Problem 37: conflicting requests 2024-04-12T21:42:13.526 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.526 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.526 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.526 INFO:teuthology.orchestra.run.smithi114.stderr: Problem 38: cannot install the best candidate for the job 2024-04-12T21:42:13.526 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.526 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.526 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.526 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.526 INFO:teuthology.orchestra.run.smithi114.stderr: Problem 39: conflicting requests 2024-04-12T21:42:13.526 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.526 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.526 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.527 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.527 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.527 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.527 INFO:teuthology.orchestra.run.smithi114.stderr: Problem 40: cannot install the best candidate for the job 2024-04-12T21:42:13.527 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.527 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.527 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.527 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.527 INFO:teuthology.orchestra.run.smithi114.stderr: Problem 41: cannot install the best candidate for the job 2024-04-12T21:42:13.527 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.527 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.527 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.527 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.528 INFO:teuthology.orchestra.run.smithi114.stderr: Problem 42: cannot install the best candidate for the job 2024-04-12T21:42:13.528 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.528 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.528 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.528 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.528 INFO:teuthology.orchestra.run.smithi114.stderr: Problem 43: cannot install the best candidate for the job 2024-04-12T21:42:13.528 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.528 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.528 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.528 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.528 INFO:teuthology.orchestra.run.smithi114.stderr: Problem 44: cannot install the best candidate for the job 2024-04-12T21:42:13.528 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.528 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.528 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.528 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.529 INFO:teuthology.orchestra.run.smithi114.stderr: Problem 45: cannot install the best candidate for the job 2024-04-12T21:42:13.529 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.529 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.529 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.529 INFO:teuthology.orchestra.run.smithi114.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-12T21:42:13.719 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-04-12T21:42:13.930 INFO:teuthology.orchestra.run.smithi114.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-12T21:42:13.930 INFO:teuthology.orchestra.run.smithi114.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-12T21:42:14.044 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-04-12T21:42:14.055 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-04-12T21:42:14.055 INFO:teuthology.orchestra.run.smithi114.stdout: Package Architecture Version Repository Size 2024-04-12T21:42:14.055 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-04-12T21:42:14.055 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-04-12T21:42:14.055 INFO:teuthology.orchestra.run.smithi114.stdout: kernel x86_64 5.14.0-435.el9 baseos 214 k 2024-04-12T21:42:14.055 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-04-12T21:42:14.055 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-core x86_64 5.14.0-435.el9 baseos 15 M 2024-04-12T21:42:14.055 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-modules x86_64 5.14.0-435.el9 baseos 34 M 2024-04-12T21:42:14.055 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-modules-core x86_64 5.14.0-435.el9 baseos 28 M 2024-04-12T21:42:14.055 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-12T21:42:14.055 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-04-12T21:42:14.055 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-04-12T21:42:14.056 INFO:teuthology.orchestra.run.smithi114.stdout:Install 4 Packages 2024-04-12T21:42:14.056 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-12T21:42:14.056 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 77 M 2024-04-12T21:42:14.056 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 123 M 2024-04-12T21:42:14.056 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-04-12T21:42:14.073 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-04-12T21:42:14.073 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-04-12T21:42:14.258 INFO:teuthology.orchestra.run.smithi114.stdout:(1/4): kernel-5.14.0-435.el9.x86_64.rpm 2.4 MB/s | 214 kB 00:00 2024-04-12T21:42:14.933 INFO:teuthology.orchestra.run.smithi114.stdout:(2/4): kernel-core-5.14.0-435.el9.x86_64.rpm 20 MB/s | 15 MB 00:00 2024-04-12T21:42:15.107 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-04-12T21:42:15.107 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-04-12T21:42:15.550 INFO:teuthology.orchestra.run.smithi114.stdout:(3/4): kernel-modules-5.14.0-435.el9.x86_64.rpm 24 MB/s | 34 MB 00:01 2024-04-12T21:42:15.834 INFO:teuthology.orchestra.run.smithi114.stdout:(4/4): kernel-modules-core-5.14.0-435.el9.x86_6 18 MB/s | 28 MB 00:01 2024-04-12T21:42:15.834 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-04-12T21:42:15.834 INFO:teuthology.orchestra.run.smithi114.stdout:Total 43 MB/s | 77 MB 00:01 2024-04-12T21:42:16.508 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-04-12T21:42:16.841 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-04-12T21:42:16.841 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-04-12T21:42:17.575 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-04-12T21:42:17.897 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-04-12T21:42:17.898 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-04-12T21:42:18.151 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : kernel-modules-core-5.14.0-435.el9.x86_64 1/4 2024-04-12T21:42:18.244 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : kernel-core-5.14.0-435.el9.x86_64 2/4 2024-04-12T21:42:20.347 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-04-12T21:42:20.951 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-modules-core-5.14.0-435.el9.x86_64 1/4 2024-04-12T21:42:21.019 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: kernel-core-5.14.0-435.el9.x86_64 2/4 2024-04-12T21:42:21.042 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-core-5.14.0-435.el9.x86_64 2/4 2024-04-12T21:42:21.147 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : kernel-modules-5.14.0-435.el9.x86_64 3/4 2024-04-12T21:42:23.782 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-core-5.14.0-435.el9.x86_64 2/4 2024-04-12T21:42:23.918 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-modules-5.14.0-435.el9.x86_64 3/4 2024-04-12T21:42:25.881 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: kernel-modules-5.14.0-435.el9.x86_64 3/4 2024-04-12T21:42:25.990 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : kernel-5.14.0-435.el9.x86_64 4/4 2024-04-12T21:42:28.628 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-modules-5.14.0-435.el9.x86_64 3/4 2024-04-12T21:42:28.794 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-5.14.0-435.el9.x86_64 4/4 2024-04-12T21:42:30.743 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: kernel-modules-core-5.14.0-435.el9.x86_64 4/4 2024-04-12T21:42:33.530 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-modules-core-5.14.0-435.el9.x86_64 4/4 2024-04-12T21:43:00.175 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: kernel-core-5.14.0-435.el9.x86_64 4/4 2024-04-12T21:43:00.176 INFO:teuthology.orchestra.run.smithi097.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-12T21:43:00.176 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-12T21:43:00.220 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: kernel-modules-5.14.0-435.el9.x86_64 4/4 2024-04-12T21:43:00.579 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: kernel-5.14.0-435.el9.x86_64 4/4 2024-04-12T21:43:00.579 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : kernel-5.14.0-435.el9.x86_64 1/4 2024-04-12T21:43:00.579 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : kernel-core-5.14.0-435.el9.x86_64 2/4 2024-04-12T21:43:00.579 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : kernel-modules-5.14.0-435.el9.x86_64 3/4 2024-04-12T21:43:01.054 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : kernel-modules-core-5.14.0-435.el9.x86_64 4/4 2024-04-12T21:43:01.054 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-12T21:43:01.054 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-04-12T21:43:01.054 INFO:teuthology.orchestra.run.smithi097.stdout: kernel-5.14.0-435.el9.x86_64 2024-04-12T21:43:01.054 INFO:teuthology.orchestra.run.smithi097.stdout: kernel-core-5.14.0-435.el9.x86_64 2024-04-12T21:43:01.054 INFO:teuthology.orchestra.run.smithi097.stdout: kernel-modules-5.14.0-435.el9.x86_64 2024-04-12T21:43:01.054 INFO:teuthology.orchestra.run.smithi097.stdout: kernel-modules-core-5.14.0-435.el9.x86_64 2024-04-12T21:43:01.054 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-12T21:43:01.055 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-04-12T21:43:01.364 DEBUG:teuthology.orchestra.run.smithi097:> rpm -q kernel | sort -rV | head -n 1 2024-04-12T21:43:01.471 INFO:teuthology.orchestra.run.smithi097.stdout:kernel-5.14.0-435.el9.x86_64 2024-04-12T21:43:01.472 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-435.el9.x86_64 2024-04-12T21:43:01.472 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-435.el9.x86_64 2024-04-12T21:43:01.472 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-12T21:43:01.472 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-12T21:43:01.472 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-12T21:43:01.472 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi097.front.sepia.ceph.com, path=None, version=distro) 2024-04-12T21:43:01.472 DEBUG:teuthology.orchestra.run.smithi097:> rpm -q kernel | sort -rV | head -n 1 2024-04-12T21:43:01.553 INFO:teuthology.orchestra.run.smithi097.stdout:kernel-5.14.0-435.el9.x86_64 2024-04-12T21:43:01.553 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-435.el9.x86_64 2024-04-12T21:43:01.553 DEBUG:teuthology.orchestra.run.smithi097:> sudo rpm -qi grub2-tools 2024-04-12T21:43:01.587 INFO:teuthology.orchestra.run.smithi097.stdout:Name : grub2-tools 2024-04-12T21:43:01.587 INFO:teuthology.orchestra.run.smithi097.stdout:Epoch : 1 2024-04-12T21:43:01.587 INFO:teuthology.orchestra.run.smithi097.stdout:Version : 2.06 2024-04-12T21:43:01.587 INFO:teuthology.orchestra.run.smithi097.stdout:Release : 68.el9 2024-04-12T21:43:01.587 INFO:teuthology.orchestra.run.smithi097.stdout:Architecture: x86_64 2024-04-12T21:43:01.587 INFO:teuthology.orchestra.run.smithi097.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-12T21:43:01.587 INFO:teuthology.orchestra.run.smithi097.stdout:Group : Unspecified 2024-04-12T21:43:01.587 INFO:teuthology.orchestra.run.smithi097.stdout:Size : 8264254 2024-04-12T21:43:01.587 INFO:teuthology.orchestra.run.smithi097.stdout:License : GPLv3+ 2024-04-12T21:43:01.588 INFO:teuthology.orchestra.run.smithi097.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-12T21:43:01.588 INFO:teuthology.orchestra.run.smithi097.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-12T21:43:01.588 INFO:teuthology.orchestra.run.smithi097.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-12T21:43:01.588 INFO:teuthology.orchestra.run.smithi097.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-12T21:43:01.588 INFO:teuthology.orchestra.run.smithi097.stdout:Packager : builder@centos.org 2024-04-12T21:43:01.588 INFO:teuthology.orchestra.run.smithi097.stdout:Vendor : CentOS 2024-04-12T21:43:01.588 INFO:teuthology.orchestra.run.smithi097.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-12T21:43:01.588 INFO:teuthology.orchestra.run.smithi097.stdout:Summary : Support tools for GRUB. 2024-04-12T21:43:01.588 INFO:teuthology.orchestra.run.smithi097.stdout:Description : 2024-04-12T21:43:01.588 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-12T21:43:01.588 INFO:teuthology.orchestra.run.smithi097.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-12T21:43:01.588 INFO:teuthology.orchestra.run.smithi097.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-12T21:43:01.588 INFO:teuthology.orchestra.run.smithi097.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-12T21:43:01.588 INFO:teuthology.orchestra.run.smithi097.stdout:hardware devices. 2024-04-12T21:43:01.588 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-12T21:43:01.589 INFO:teuthology.orchestra.run.smithi097.stdout:This subpackage provides tools for support of all platforms. 2024-04-12T21:43:01.589 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-12T21:43:01.590 INFO:teuthology.task.kernel:Updating grub on smithi097 to boot 5.14.0-435.el9.x86_64 2024-04-12T21:43:01.590 DEBUG:teuthology.orchestra.run.smithi097:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-12T21:43:02.212 INFO:teuthology.orchestra.run.smithi097.stderr:Generating grub configuration file ... 2024-04-12T21:43:02.775 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-core-5.14.0-435.el9.x86_64 4/4 2024-04-12T21:43:02.776 INFO:teuthology.orchestra.run.smithi114.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-12T21:43:02.776 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-12T21:43:02.821 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-modules-5.14.0-435.el9.x86_64 4/4 2024-04-12T21:43:03.167 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-5.14.0-435.el9.x86_64 4/4 2024-04-12T21:43:03.168 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-5.14.0-435.el9.x86_64 1/4 2024-04-12T21:43:03.168 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-core-5.14.0-435.el9.x86_64 2/4 2024-04-12T21:43:03.168 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-modules-5.14.0-435.el9.x86_64 3/4 2024-04-12T21:43:03.559 INFO:teuthology.orchestra.run.smithi097.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-12T21:43:03.598 INFO:teuthology.orchestra.run.smithi097.stderr:done 2024-04-12T21:43:03.600 DEBUG:teuthology.orchestra.run.smithi097:> mktemp 2024-04-12T21:43:03.615 INFO:teuthology.orchestra.run.smithi097.stdout:/tmp/tmp.whGbnATqrX 2024-04-12T21:43:03.615 DEBUG:teuthology.orchestra.run.smithi097:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.whGbnATqrX 2024-04-12T21:43:03.616 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-modules-core-5.14.0-435.el9.x86_64 4/4 2024-04-12T21:43:03.616 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-12T21:43:03.616 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-04-12T21:43:03.616 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-5.14.0-435.el9.x86_64 2024-04-12T21:43:03.616 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-core-5.14.0-435.el9.x86_64 2024-04-12T21:43:03.616 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-modules-5.14.0-435.el9.x86_64 2024-04-12T21:43:03.616 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-modules-core-5.14.0-435.el9.x86_64 2024-04-12T21:43:03.616 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-12T21:43:03.616 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-04-12T21:43:03.685 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 0666 /tmp/tmp.whGbnATqrX 2024-04-12T21:43:03.821 DEBUG:teuthology.orchestra.remote:smithi097:/tmp/tmp.whGbnATqrX is 6KB 2024-04-12T21:43:03.869 DEBUG:teuthology.orchestra.run.smithi097:> rm -fr /tmp/tmp.whGbnATqrX 2024-04-12T21:43:03.883 DEBUG:teuthology.orchestra.run.smithi097:> sudo /bin/ls /boot/loader/entries || true 2024-04-12T21:43:03.950 DEBUG:teuthology.orchestra.run.smithi114:> rpm -q kernel | sort -rV | head -n 1 2024-04-12T21:43:03.957 INFO:teuthology.orchestra.run.smithi097.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-12T21:43:03.957 INFO:teuthology.orchestra.run.smithi097.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-12T21:43:03.957 INFO:teuthology.orchestra.run.smithi097.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-12T21:43:03.957 INFO:teuthology.orchestra.run.smithi097.stdout:15949e3c88704300882482eddfd7baae-5.14.0-435.el9.x86_64.conf 2024-04-12T21:43:03.959 DEBUG:teuthology.orchestra.run.smithi097:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-435.el9.x86_64 2024-04-12T21:43:04.055 INFO:teuthology.orchestra.run.smithi114.stdout:kernel-5.14.0-435.el9.x86_64 2024-04-12T21:43:04.055 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-435.el9.x86_64 2024-04-12T21:43:04.056 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-435.el9.x86_64 2024-04-12T21:43:04.056 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-12T21:43:04.056 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-12T21:43:04.056 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-04-12T21:43:04.056 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi114.front.sepia.ceph.com, path=None, version=distro) 2024-04-12T21:43:04.056 DEBUG:teuthology.orchestra.run.smithi114:> rpm -q kernel | sort -rV | head -n 1 2024-04-12T21:43:04.095 DEBUG:teuthology.orchestra.run.smithi097:> sudo shutdown -r now 2024-04-12T21:43:04.136 INFO:teuthology.orchestra.run.smithi114.stdout:kernel-5.14.0-435.el9.x86_64 2024-04-12T21:43:04.136 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-435.el9.x86_64 2024-04-12T21:43:04.136 DEBUG:teuthology.orchestra.run.smithi114:> sudo rpm -qi grub2-tools 2024-04-12T21:43:04.170 INFO:teuthology.orchestra.run.smithi114.stdout:Name : grub2-tools 2024-04-12T21:43:04.170 INFO:teuthology.orchestra.run.smithi114.stdout:Epoch : 1 2024-04-12T21:43:04.170 INFO:teuthology.orchestra.run.smithi114.stdout:Version : 2.06 2024-04-12T21:43:04.170 INFO:teuthology.orchestra.run.smithi114.stdout:Release : 68.el9 2024-04-12T21:43:04.170 INFO:teuthology.orchestra.run.smithi114.stdout:Architecture: x86_64 2024-04-12T21:43:04.170 INFO:teuthology.orchestra.run.smithi114.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-12T21:43:04.170 INFO:teuthology.orchestra.run.smithi114.stdout:Group : Unspecified 2024-04-12T21:43:04.170 INFO:teuthology.orchestra.run.smithi114.stdout:Size : 8264254 2024-04-12T21:43:04.171 INFO:teuthology.orchestra.run.smithi114.stdout:License : GPLv3+ 2024-04-12T21:43:04.171 INFO:teuthology.orchestra.run.smithi114.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-12T21:43:04.171 INFO:teuthology.orchestra.run.smithi114.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-12T21:43:04.171 INFO:teuthology.orchestra.run.smithi114.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-12T21:43:04.171 INFO:teuthology.orchestra.run.smithi114.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-12T21:43:04.171 INFO:teuthology.orchestra.run.smithi114.stdout:Packager : builder@centos.org 2024-04-12T21:43:04.171 INFO:teuthology.orchestra.run.smithi114.stdout:Vendor : CentOS 2024-04-12T21:43:04.171 INFO:teuthology.orchestra.run.smithi114.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-12T21:43:04.171 INFO:teuthology.orchestra.run.smithi114.stdout:Summary : Support tools for GRUB. 2024-04-12T21:43:04.171 INFO:teuthology.orchestra.run.smithi114.stdout:Description : 2024-04-12T21:43:04.171 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-12T21:43:04.171 INFO:teuthology.orchestra.run.smithi114.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-12T21:43:04.171 INFO:teuthology.orchestra.run.smithi114.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-12T21:43:04.171 INFO:teuthology.orchestra.run.smithi114.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-12T21:43:04.172 INFO:teuthology.orchestra.run.smithi114.stdout:hardware devices. 2024-04-12T21:43:04.172 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-12T21:43:04.172 INFO:teuthology.orchestra.run.smithi114.stdout:This subpackage provides tools for support of all platforms. 2024-04-12T21:43:04.173 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-12T21:43:04.173 INFO:teuthology.task.kernel:Updating grub on smithi114 to boot 5.14.0-435.el9.x86_64 2024-04-12T21:43:04.173 DEBUG:teuthology.orchestra.run.smithi114:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-12T21:43:04.812 INFO:teuthology.orchestra.run.smithi114.stderr:Generating grub configuration file ... 2024-04-12T21:43:06.166 INFO:teuthology.orchestra.run.smithi114.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-12T21:43:06.201 INFO:teuthology.orchestra.run.smithi114.stderr:done 2024-04-12T21:43:06.204 DEBUG:teuthology.orchestra.run.smithi114:> mktemp 2024-04-12T21:43:06.219 INFO:teuthology.orchestra.run.smithi114.stdout:/tmp/tmp.WUQvTzCimo 2024-04-12T21:43:06.220 DEBUG:teuthology.orchestra.run.smithi114:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.WUQvTzCimo 2024-04-12T21:43:06.288 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0666 /tmp/tmp.WUQvTzCimo 2024-04-12T21:43:06.426 DEBUG:teuthology.orchestra.remote:smithi114:/tmp/tmp.WUQvTzCimo is 6KB 2024-04-12T21:43:06.475 DEBUG:teuthology.orchestra.run.smithi114:> rm -fr /tmp/tmp.WUQvTzCimo 2024-04-12T21:43:06.489 DEBUG:teuthology.orchestra.run.smithi114:> sudo /bin/ls /boot/loader/entries || true 2024-04-12T21:43:06.563 INFO:teuthology.orchestra.run.smithi114.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-12T21:43:06.563 INFO:teuthology.orchestra.run.smithi114.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-12T21:43:06.563 INFO:teuthology.orchestra.run.smithi114.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-12T21:43:06.563 INFO:teuthology.orchestra.run.smithi114.stdout:15949e3c88704300882482eddfd7baae-5.14.0-435.el9.x86_64.conf 2024-04-12T21:43:06.565 DEBUG:teuthology.orchestra.run.smithi114:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-435.el9.x86_64 2024-04-12T21:43:06.700 DEBUG:teuthology.orchestra.run.smithi114:> sudo shutdown -r now 2024-04-12T21:43:34.122 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-12T21:43:34.123 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-04-12T21:43:34.124 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-12T21:43:36.704 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-04-12T21:43:36.704 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-04-12T21:43:36.704 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-12T21:44:08.735 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.97 2024-04-12T21:44:11.294 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.114 2024-04-12T21:44:17.742 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-04-12T21:44:17.743 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-12T21:44:20.298 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-04-12T21:44:20.299 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-12T21:44:36.126 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.97 2024-04-12T21:44:38.690 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.114 2024-04-12T21:44:48.137 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-04-12T21:44:48.138 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-12T21:44:48.539 DEBUG:teuthology.orchestra.run.smithi097:> true 2024-04-12T21:44:48.913 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-04-12T21:44:48.913 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-435.el9.x86_64"... 2024-04-12T21:44:48.913 DEBUG:teuthology.orchestra.run.smithi097:> uname -r 2024-04-12T21:44:48.927 INFO:teuthology.orchestra.run.smithi097.stdout:5.14.0-435.el9.x86_64 2024-04-12T21:44:48.928 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-435.el9.x86_64 vs 5.14.0-435.el9.x86_64 2024-04-12T21:44:48.928 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-12T21:44:48.928 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-12T21:44:49.929 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-12T21:44:49.929 DEBUG:teuthology.orchestra.run.smithi097:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-12T21:44:49.997 INFO:teuthology.orchestra.run.smithi097.stdout:ttyS1 2024-04-12T21:44:50.034 DEBUG:teuthology.parallel:result is None 2024-04-12T21:44:50.692 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-04-12T21:44:50.692 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-12T21:44:51.206 DEBUG:teuthology.orchestra.run.smithi114:> true 2024-04-12T21:44:51.621 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-04-12T21:44:51.621 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-435.el9.x86_64"... 2024-04-12T21:44:51.621 DEBUG:teuthology.orchestra.run.smithi114:> uname -r 2024-04-12T21:44:51.635 INFO:teuthology.orchestra.run.smithi114.stdout:5.14.0-435.el9.x86_64 2024-04-12T21:44:51.636 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-435.el9.x86_64 vs 5.14.0-435.el9.x86_64 2024-04-12T21:44:51.636 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-12T21:44:51.636 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-12T21:44:52.636 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-12T21:44:52.637 DEBUG:teuthology.orchestra.run.smithi114:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-12T21:44:52.721 INFO:teuthology.orchestra.run.smithi114.stdout:ttyS1 2024-04-12T21:44:52.756 DEBUG:teuthology.parallel:result is None 2024-04-12T21:44:52.756 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-12T21:44:52.763 INFO:teuthology.task.internal:Creating test directory... 2024-04-12T21:44:52.763 DEBUG:teuthology.orchestra.run.smithi097:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-12T21:44:52.768 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-12T21:44:52.794 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-12T21:44:52.917 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-12T21:44:52.924 INFO:teuthology.task.internal:Creating archive directory... 2024-04-12T21:44:52.924 DEBUG:teuthology.orchestra.run.smithi097:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-12T21:44:52.926 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-12T21:44:52.976 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-12T21:44:52.983 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-12T21:44:52.983 DEBUG:teuthology.orchestra.run.smithi097:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-12T21:44:53.017 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-12T21:44:53.050 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T21:44:53.063 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T21:44:53.113 INFO:teuthology.orchestra.run.smithi097.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T21:44:53.140 INFO:teuthology.orchestra.run.smithi097.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T21:44:53.143 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-12T21:44:53.150 INFO:teuthology.task.internal:Configuring sudo... 2024-04-12T21:44:53.150 DEBUG:teuthology.orchestra.run.smithi097:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-12T21:44:53.155 DEBUG:teuthology.orchestra.run.smithi114:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-12T21:44:53.197 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-12T21:44:53.206 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-12T21:44:53.206 DEBUG:teuthology.orchestra.run.smithi097:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-12T21:44:53.244 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-12T21:44:53.268 DEBUG:teuthology.orchestra.run.smithi097:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-12T21:44:53.368 DEBUG:teuthology.orchestra.run.smithi097:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-12T21:44:53.445 DEBUG:teuthology.orchestra.run.smithi097:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-12T21:44:53.499 DEBUG:teuthology.orchestra.run.smithi097:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-12T21:44:53.566 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-12T21:44:53.566 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-12T21:44:53.652 DEBUG:teuthology.orchestra.run.smithi114:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-12T21:44:53.699 DEBUG:teuthology.orchestra.run.smithi114:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-12T21:44:53.767 DEBUG:teuthology.orchestra.run.smithi114:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-12T21:44:53.800 DEBUG:teuthology.orchestra.run.smithi114:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-12T21:44:53.849 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-04-12T21:44:53.849 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-12T21:44:53.915 DEBUG:teuthology.orchestra.run.smithi097:> sudo service rsyslog restart 2024-04-12T21:44:53.917 DEBUG:teuthology.orchestra.run.smithi114:> sudo service rsyslog restart 2024-04-12T21:44:53.958 INFO:teuthology.orchestra.run.smithi097.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-12T21:44:54.050 INFO:teuthology.orchestra.run.smithi114.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-12T21:44:54.300 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-12T21:44:54.307 INFO:teuthology.task.internal:Starting timer... 2024-04-12T21:44:54.307 INFO:teuthology.run_tasks:Running task pcp... 2024-04-12T21:44:54.318 INFO:teuthology.run_tasks:Running task selinux... 2024-04-12T21:44:54.326 DEBUG:teuthology.orchestra.run.smithi097:> sudo service auditd rotate 2024-04-12T21:44:54.459 INFO:teuthology.orchestra.run.smithi097.stdout:Rotating logs: 2024-04-12T21:44:54.461 DEBUG:teuthology.orchestra.run.smithi114:> sudo service auditd rotate 2024-04-12T21:44:54.545 INFO:teuthology.orchestra.run.smithi114.stdout:Rotating logs: 2024-04-12T21:44:54.553 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-12T21:44:54.553 DEBUG:teuthology.orchestra.run.smithi097:> /usr/sbin/getenforce 2024-04-12T21:44:54.576 INFO:teuthology.orchestra.run.smithi097.stdout:Permissive 2024-04-12T21:44:54.576 DEBUG:teuthology.orchestra.run.smithi114:> /usr/sbin/getenforce 2024-04-12T21:44:54.609 INFO:teuthology.orchestra.run.smithi114.stdout:Permissive 2024-04-12T21:44:54.610 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi097.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi114.front.sepia.ceph.com': 'permissive'} 2024-04-12T21:44:54.610 DEBUG:teuthology.orchestra.run.smithi097:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-12T21:44:54.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-12T21:44:54.647 DEBUG:teuthology.orchestra.run.smithi114:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-12T21:44:54.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-12T21:44:54.706 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-12T21:44:54.706 DEBUG:teuthology.orchestra.run.smithi097:> sudo /usr/sbin/setenforce permissive 2024-04-12T21:44:54.736 DEBUG:teuthology.orchestra.run.smithi114:> sudo /usr/sbin/setenforce permissive 2024-04-12T21:44:54.987 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-12T21:44:54.997 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-12T21:44:55.014 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-12T21:44:55.016 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi097.front.sepia.ceph.com,smithi114.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-12T21:49:13.312 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi097.front.sepia.ceph.com'), Remote(name='ubuntu@smithi114.front.sepia.ceph.com')] 2024-04-12T21:49:13.313 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-04-12T21:49:13.314 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-12T21:49:13.404 DEBUG:teuthology.orchestra.run.smithi097:> true 2024-04-12T21:49:13.481 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-04-12T21:49:13.481 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-04-12T21:49:13.482 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-12T21:49:13.568 DEBUG:teuthology.orchestra.run.smithi114:> true 2024-04-12T21:49:13.646 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-04-12T21:49:13.646 INFO:teuthology.run_tasks:Running task clock... 2024-04-12T21:49:13.656 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-12T21:49:13.657 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-12T21:49:13.657 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-12T21:49:13.661 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-12T21:49:13.661 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-12T21:49:13.689 INFO:teuthology.orchestra.run.smithi097.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-12T21:49:13.706 INFO:teuthology.orchestra.run.smithi097.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-12T21:49:13.718 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-12T21:49:13.734 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-12T21:49:13.735 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: ntpd: command not found 2024-04-12T21:49:13.749 INFO:teuthology.orchestra.run.smithi097.stdout:506 Cannot talk to daemon 2024-04-12T21:49:13.763 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: ntpd: command not found 2024-04-12T21:49:13.765 INFO:teuthology.orchestra.run.smithi097.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-12T21:49:13.777 INFO:teuthology.orchestra.run.smithi114.stdout:506 Cannot talk to daemon 2024-04-12T21:49:13.781 INFO:teuthology.orchestra.run.smithi097.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-12T21:49:13.793 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-12T21:49:13.810 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-12T21:49:13.837 INFO:teuthology.orchestra.run.smithi097.stderr:bash: line 1: ntpq: command not found 2024-04-12T21:49:13.840 INFO:teuthology.orchestra.run.smithi097.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-12T21:49:13.840 INFO:teuthology.orchestra.run.smithi097.stdout:=============================================================================== 2024-04-12T21:49:13.840 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T21:49:13.840 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T21:49:13.840 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T21:49:13.840 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T21:49:13.863 INFO:teuthology.orchestra.run.smithi114.stderr:bash: line 1: ntpq: command not found 2024-04-12T21:49:13.867 INFO:teuthology.orchestra.run.smithi114.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-12T21:49:13.868 INFO:teuthology.orchestra.run.smithi114.stdout:=============================================================================== 2024-04-12T21:49:13.868 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T21:49:13.868 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T21:49:13.868 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T21:49:13.868 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T21:49:13.868 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-12T21:49:13.968 INFO:tasks.cephadm:Config: {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'conf': {'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': False}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'b277feb2c7beb4597d8ecc39d58d5c36b81f613a'} 2024-04-12T21:49:13.968 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v17.2.0 2024-04-12T21:49:13.969 INFO:tasks.cephadm:Cluster fsid is 80bf7680-f916-11ee-bc8d-c7b262605968 2024-04-12T21:49:13.969 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-12T21:49:13.969 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.97', 'mon.c': '[v2:172.21.15.97:3301,v1:172.21.15.97:6790]', 'mon.b': '172.21.15.114'} 2024-04-12T21:49:13.969 INFO:tasks.cephadm:First mon is mon.a on smithi097 2024-04-12T21:49:13.969 INFO:tasks.cephadm:First mgr is y 2024-04-12T21:49:13.969 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-12T21:49:13.970 DEBUG:teuthology.orchestra.run.smithi097:> sudo hostname $(hostname -s) 2024-04-12T21:49:13.997 DEBUG:teuthology.orchestra.run.smithi114:> sudo hostname $(hostname -s) 2024-04-12T21:49:14.024 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v17.2.0)... 2024-04-12T21:49:14.025 DEBUG:teuthology.orchestra.run.smithi097:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-12T21:49:14.299 INFO:teuthology.orchestra.run.smithi097.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Apr 12 21:49 /home/ubuntu/cephtest/cephadm 2024-04-12T21:49:14.299 DEBUG:teuthology.orchestra.run.smithi114:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-12T21:49:14.579 INFO:teuthology.orchestra.run.smithi114.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Apr 12 21:49 /home/ubuntu/cephtest/cephadm 2024-04-12T21:49:14.580 DEBUG:teuthology.orchestra.run.smithi097:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-12T21:49:14.596 DEBUG:teuthology.orchestra.run.smithi114:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-12T21:49:14.622 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v17.2.0 on all hosts... 2024-04-12T21:49:14.622 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-04-12T21:49:14.638 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-04-12T21:49:15.546 INFO:teuthology.orchestra.run.smithi114.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-12T21:49:15.581 INFO:teuthology.orchestra.run.smithi097.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-12T21:49:34.386 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-04-12T21:49:34.386 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-04-12T21:49:34.387 INFO:teuthology.orchestra.run.smithi114.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-04-12T21:49:34.387 INFO:teuthology.orchestra.run.smithi114.stdout: "repo_digests": [ 2024-04-12T21:49:34.387 INFO:teuthology.orchestra.run.smithi114.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-04-12T21:49:34.387 INFO:teuthology.orchestra.run.smithi114.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-04-12T21:49:34.387 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-12T21:49:34.387 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-04-12T21:49:39.119 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-12T21:49:39.119 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-04-12T21:49:39.119 INFO:teuthology.orchestra.run.smithi097.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-04-12T21:49:39.119 INFO:teuthology.orchestra.run.smithi097.stdout: "repo_digests": [ 2024-04-12T21:49:39.119 INFO:teuthology.orchestra.run.smithi097.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-04-12T21:49:39.120 INFO:teuthology.orchestra.run.smithi097.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-04-12T21:49:39.120 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-12T21:49:39.120 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-12T21:49:39.137 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /etc/ceph 2024-04-12T21:49:39.189 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /etc/ceph 2024-04-12T21:49:39.225 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 777 /etc/ceph 2024-04-12T21:49:39.251 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 777 /etc/ceph 2024-04-12T21:49:39.296 INFO:tasks.cephadm:Writing seed config... 2024-04-12T21:49:39.297 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-04-12T21:49:39.297 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-12T21:49:39.298 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-12T21:49:39.298 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = False 2024-04-12T21:49:39.298 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-12T21:49:39.298 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-12T21:49:39.298 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-12T21:49:39.298 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-12T21:49:39.298 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-12T21:49:39.299 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-12T21:49:39.299 DEBUG:teuthology.orchestra.run.smithi097:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-12T21:49:39.314 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 80bf7680-f916-11ee-bc8d-c7b262605968 mon election default strategy = 1 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = False [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-04-12T21:49:39.314 DEBUG:teuthology.orchestra.run.smithi097:mon.a> sudo journalctl -f -n 0 -u ceph-80bf7680-f916-11ee-bc8d-c7b262605968@mon.a.service 2024-04-12T21:49:39.357 DEBUG:teuthology.orchestra.run.smithi097:mgr.y> sudo journalctl -f -n 0 -u ceph-80bf7680-f916-11ee-bc8d-c7b262605968@mgr.y.service 2024-04-12T21:49:39.400 INFO:tasks.cephadm:Bootstrapping... 2024-04-12T21:49:39.400 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 -v bootstrap --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.97 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-12T21:49:39.594 INFO:teuthology.orchestra.run.smithi097.stderr:-------------------------------------------------------------------------------- 2024-04-12T21:49:39.594 INFO:teuthology.orchestra.run.smithi097.stderr:cephadm ['--image', 'quay.io/ceph/ceph:v17.2.0', '-v', 'bootstrap', '--fsid', '80bf7680-f916-11ee-bc8d-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.97', '--skip-admin-label'] 2024-04-12T21:49:39.617 INFO:teuthology.orchestra.run.smithi097.stderr:/bin/podman: 4.9.4-dev 2024-04-12T21:49:39.619 INFO:teuthology.orchestra.run.smithi097.stderr:Verifying podman|docker is present... 2024-04-12T21:49:39.640 INFO:teuthology.orchestra.run.smithi097.stderr:/bin/podman: 4.9.4-dev 2024-04-12T21:49:39.642 INFO:teuthology.orchestra.run.smithi097.stderr:Verifying lvm2 is present... 2024-04-12T21:49:39.642 INFO:teuthology.orchestra.run.smithi097.stderr:Verifying time synchronization is in place... 2024-04-12T21:49:39.648 INFO:teuthology.orchestra.run.smithi097.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-04-12T21:49:39.654 INFO:teuthology.orchestra.run.smithi097.stderr:systemctl: inactive 2024-04-12T21:49:39.661 INFO:teuthology.orchestra.run.smithi097.stderr:systemctl: enabled 2024-04-12T21:49:39.667 INFO:teuthology.orchestra.run.smithi097.stderr:systemctl: active 2024-04-12T21:49:39.668 INFO:teuthology.orchestra.run.smithi097.stderr:Unit chronyd.service is enabled and running 2024-04-12T21:49:39.668 INFO:teuthology.orchestra.run.smithi097.stderr:Repeating the final host check... 2024-04-12T21:49:39.689 INFO:teuthology.orchestra.run.smithi097.stderr:/bin/podman: 4.9.4-dev 2024-04-12T21:49:39.691 INFO:teuthology.orchestra.run.smithi097.stderr:podman (/bin/podman) version 4.9.4 is present 2024-04-12T21:49:39.691 INFO:teuthology.orchestra.run.smithi097.stderr:systemctl is present 2024-04-12T21:49:39.691 INFO:teuthology.orchestra.run.smithi097.stderr:lvcreate is present 2024-04-12T21:49:39.697 INFO:teuthology.orchestra.run.smithi097.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-04-12T21:49:39.703 INFO:teuthology.orchestra.run.smithi097.stderr:systemctl: inactive 2024-04-12T21:49:39.709 INFO:teuthology.orchestra.run.smithi097.stderr:systemctl: enabled 2024-04-12T21:49:39.715 INFO:teuthology.orchestra.run.smithi097.stderr:systemctl: active 2024-04-12T21:49:39.716 INFO:teuthology.orchestra.run.smithi097.stderr:Unit chronyd.service is enabled and running 2024-04-12T21:49:39.716 INFO:teuthology.orchestra.run.smithi097.stderr:Host looks OK 2024-04-12T21:49:39.716 INFO:teuthology.orchestra.run.smithi097.stderr:Cluster fsid: 80bf7680-f916-11ee-bc8d-c7b262605968 2024-04-12T21:49:39.716 INFO:teuthology.orchestra.run.smithi097.stderr:Acquiring lock 140017251454304 on /run/cephadm/80bf7680-f916-11ee-bc8d-c7b262605968.lock 2024-04-12T21:49:39.716 INFO:teuthology.orchestra.run.smithi097.stderr:Lock 140017251454304 acquired on /run/cephadm/80bf7680-f916-11ee-bc8d-c7b262605968.lock 2024-04-12T21:49:39.717 INFO:teuthology.orchestra.run.smithi097.stderr:Verifying IP 172.21.15.97 port 3300 ... 2024-04-12T21:49:39.717 INFO:teuthology.orchestra.run.smithi097.stderr:Verifying IP 172.21.15.97 port 6789 ... 2024-04-12T21:49:39.717 INFO:teuthology.orchestra.run.smithi097.stderr:Base mon IP is 172.21.15.97, final addrv is [v2:172.21.15.97:3300,v1:172.21.15.97:6789] 2024-04-12T21:49:39.719 INFO:teuthology.orchestra.run.smithi097.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.97 metric 100 2024-04-12T21:49:39.719 INFO:teuthology.orchestra.run.smithi097.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.97 metric 100 2024-04-12T21:49:39.722 INFO:teuthology.orchestra.run.smithi097.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-04-12T21:49:39.722 INFO:teuthology.orchestra.run.smithi097.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-04-12T21:49:39.722 INFO:teuthology.orchestra.run.smithi097.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1794sec hoplimit 64 pref medium 2024-04-12T21:49:39.724 INFO:teuthology.orchestra.run.smithi097.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-12T21:49:39.724 INFO:teuthology.orchestra.run.smithi097.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-04-12T21:49:39.725 INFO:teuthology.orchestra.run.smithi097.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-04-12T21:49:39.725 INFO:teuthology.orchestra.run.smithi097.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-04-12T21:49:39.725 INFO:teuthology.orchestra.run.smithi097.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe8f:d805/64 scope link 2024-04-12T21:49:39.725 INFO:teuthology.orchestra.run.smithi097.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-04-12T21:49:39.725 INFO:teuthology.orchestra.run.smithi097.stderr:Mon IP `172.21.15.97` is in CIDR network `172.21.0.0/20` 2024-04-12T21:49:39.725 INFO:teuthology.orchestra.run.smithi097.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-12T21:49:39.726 INFO:teuthology.orchestra.run.smithi097.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-12T21:49:39.827 INFO:teuthology.orchestra.run.smithi097.stderr:/bin/podman: Trying to pull quay.io/ceph/ceph:v17.2.0... 2024-04-12T21:49:40.201 INFO:teuthology.orchestra.run.smithi097.stderr:/bin/podman: Getting image source signatures 2024-04-12T21:49:40.201 INFO:teuthology.orchestra.run.smithi097.stderr:/bin/podman: Copying blob sha256:33ca8fff7868c4dc0c11e09bca97c720eb9cfbab7221216754367dd8de70388a 2024-04-12T21:49:40.201 INFO:teuthology.orchestra.run.smithi097.stderr:/bin/podman: Copying blob sha256:89b4a75bc2d8500f15463747507c9623df43886c134463e7f0527e70900e7a7b 2024-04-12T21:49:40.201 INFO:teuthology.orchestra.run.smithi097.stderr:/bin/podman: Copying blob sha256:c32ab78b488d0c72f64eded765c0cf6b5bf2c75dab66cb62a9d367fa6ec42513 2024-04-12T21:49:40.201 INFO:teuthology.orchestra.run.smithi097.stderr:/bin/podman: Copying blob sha256:599d07cb321ff0a3c82224e1138fc685793fa69b93ed5780415751a5f7e4b8c2 2024-04-12T21:49:40.201 INFO:teuthology.orchestra.run.smithi097.stderr:/bin/podman: Copying blob sha256:a70843738bb77e1ab9c1f85969ebdfa55f178e746be081d1cb4f94011f69eb7c 2024-04-12T21:49:40.202 INFO:teuthology.orchestra.run.smithi097.stderr:/bin/podman: Copying config sha256:e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-04-12T21:49:40.381 INFO:teuthology.orchestra.run.smithi097.stderr:/bin/podman: Writing manifest to image destination 2024-04-12T21:49:40.517 INFO:teuthology.orchestra.run.smithi097.stderr:/bin/podman: e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-04-12T21:49:40.988 INFO:teuthology.orchestra.run.smithi097.stderr:ceph: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-04-12T21:49:41.541 INFO:teuthology.orchestra.run.smithi097.stderr:Ceph version: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-04-12T21:49:41.541 INFO:teuthology.orchestra.run.smithi097.stderr:Extracting ceph user uid/gid from container image... 2024-04-12T21:49:41.949 INFO:teuthology.orchestra.run.smithi097.stderr:stat: 167 167 2024-04-12T21:49:42.506 INFO:teuthology.orchestra.run.smithi097.stderr:Creating initial keys... 2024-04-12T21:49:42.913 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-authtool: AQB2rBlmCKBiMRAA5DVTdmGRUzyhQL2YNepY+Q== 2024-04-12T21:49:43.828 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-authtool: AQB3rBlm1u9ALBAAOUz6eK2Tk66gAffQWw5LEw== 2024-04-12T21:49:44.766 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph-authtool: AQB4rBlmzzMlKBAAUwFpwY2m4DsViB6oKDlhOw== 2024-04-12T21:49:45.207 INFO:teuthology.orchestra.run.smithi097.stderr:Creating initial monmap... 2024-04-12T21:49:45.614 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-12T21:49:45.614 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/monmaptool: setting min_mon_release = octopus 2024-04-12T21:49:45.614 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 80bf7680-f916-11ee-bc8d-c7b262605968 2024-04-12T21:49:45.614 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-12T21:49:46.162 INFO:teuthology.orchestra.run.smithi097.stderr:monmaptool for a [v2:172.21.15.97:3300,v1:172.21.15.97:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-12T21:49:46.162 INFO:teuthology.orchestra.run.smithi097.stderr:setting min_mon_release = octopus 2024-04-12T21:49:46.162 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/monmaptool: set fsid to 80bf7680-f916-11ee-bc8d-c7b262605968 2024-04-12T21:49:46.162 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-12T21:49:46.162 INFO:teuthology.orchestra.run.smithi097.stderr: 2024-04-12T21:49:46.162 INFO:teuthology.orchestra.run.smithi097.stderr:Creating mon... 2024-04-12T21:49:47.109 INFO:teuthology.orchestra.run.smithi097.stderr:create mon.a on 2024-04-12T21:49:47.329 INFO:teuthology.orchestra.run.smithi097.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-12T21:49:47.473 INFO:teuthology.orchestra.run.smithi097.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-80bf7680-f916-11ee-bc8d-c7b262605968.target → /etc/systemd/system/ceph-80bf7680-f916-11ee-bc8d-c7b262605968.target. 2024-04-12T21:49:47.473 INFO:teuthology.orchestra.run.smithi097.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-80bf7680-f916-11ee-bc8d-c7b262605968.target → /etc/systemd/system/ceph-80bf7680-f916-11ee-bc8d-c7b262605968.target. 2024-04-12T21:49:47.781 INFO:teuthology.orchestra.run.smithi097.stderr:systemctl: Failed to reset failed state of unit ceph-80bf7680-f916-11ee-bc8d-c7b262605968@mon.a.service: Unit ceph-80bf7680-f916-11ee-bc8d-c7b262605968@mon.a.service not loaded. 2024-04-12T21:49:47.788 INFO:teuthology.orchestra.run.smithi097.stderr:systemctl: Created symlink /etc/systemd/system/ceph-80bf7680-f916-11ee-bc8d-c7b262605968.target.wants/ceph-80bf7680-f916-11ee-bc8d-c7b262605968@mon.a.service → /etc/systemd/system/ceph-80bf7680-f916-11ee-bc8d-c7b262605968@.service. 2024-04-12T21:49:48.490 INFO:teuthology.orchestra.run.smithi097.stderr:systemctl: disabled 2024-04-12T21:49:48.496 INFO:teuthology.orchestra.run.smithi097.stderr:systemctl: inactive 2024-04-12T21:49:48.497 INFO:teuthology.orchestra.run.smithi097.stderr:firewalld.service is not enabled 2024-04-12T21:49:48.497 INFO:teuthology.orchestra.run.smithi097.stderr:Not possible to enable service . firewalld.service is not available 2024-04-12T21:49:48.498 INFO:teuthology.orchestra.run.smithi097.stderr:Waiting for mon to start... 2024-04-12T21:49:48.498 INFO:teuthology.orchestra.run.smithi097.stderr:Waiting for mon... 2024-04-12T21:49:48.995 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: cluster: 2024-04-12T21:49:48.995 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: id: 80bf7680-f916-11ee-bc8d-c7b262605968 2024-04-12T21:49:48.995 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-04-12T21:49:48.995 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: 2024-04-12T21:49:48.995 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: services: 2024-04-12T21:49:48.995 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: mon: 1 daemons, quorum a (age 0.566559s) 2024-04-12T21:49:48.995 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: mgr: no daemons active 2024-04-12T21:49:48.995 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-04-12T21:49:48.996 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: 2024-04-12T21:49:48.996 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: data: 2024-04-12T21:49:48.996 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-04-12T21:49:48.996 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-04-12T21:49:48.996 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-04-12T21:49:48.996 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: pgs: 2024-04-12T21:49:48.996 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: 2024-04-12T21:49:49.567 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:49 smithi097 ceph-mon[24494]: from='client.? 172.21.15.97:0/3071191174' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-12T21:49:49.570 INFO:teuthology.orchestra.run.smithi097.stderr:mon is available 2024-04-12T21:49:49.570 INFO:teuthology.orchestra.run.smithi097.stderr:Assimilating anything we can from ceph.conf... 2024-04-12T21:49:50.040 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: 2024-04-12T21:49:50.040 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: [global] 2024-04-12T21:49:50.040 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: fsid = 80bf7680-f916-11ee-bc8d-c7b262605968 2024-04-12T21:49:50.040 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.97:3300,v1:172.21.15.97:6789] 2024-04-12T21:49:50.040 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-04-12T21:49:50.040 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-04-12T21:49:50.040 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-04-12T21:49:50.040 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-04-12T21:49:50.040 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: 2024-04-12T21:49:50.040 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: [mgr] 2024-04-12T21:49:50.040 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: mgr/cephadm/use_agent = False 2024-04-12T21:49:50.040 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-04-12T21:49:50.040 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: 2024-04-12T21:49:50.041 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: [osd] 2024-04-12T21:49:50.041 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-04-12T21:49:50.041 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-04-12T21:49:50.576 INFO:teuthology.orchestra.run.smithi097.stderr:Generating new minimal ceph.conf... 2024-04-12T21:49:50.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:50 smithi097 ceph-mon[24494]: from='client.? 172.21.15.97:0/3845285206' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-12T21:49:50.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:50 smithi097 ceph-mon[24494]: from='client.? 172.21.15.97:0/3845285206' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-12T21:49:51.638 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:51 smithi097 ceph-mon[24494]: from='client.? 172.21.15.97:0/3344898043' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:49:51.640 INFO:teuthology.orchestra.run.smithi097.stderr:Restarting the monitor... 2024-04-12T21:49:51.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:51 smithi097 systemd[1]: Stopping Ceph mon.a for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T21:49:51.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:51 smithi097 bash[24711]: Error: no container with name or ID "ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mon.a" found: no such container 2024-04-12T21:49:51.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:51 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mon-a[24489]: 2024-04-12T21:49:51.790+0000 7f2b37168700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-12T21:49:51.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:51 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mon-a[24489]: 2024-04-12T21:49:51.790+0000 7f2b37168700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-12T21:49:52.233 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:51 smithi097 podman[24722]: 2024-04-12 21:49:51.944048457 +0000 UTC m=+0.251476388 container died 423ca0f5592d87d69094ccca4b05b2925f9576a7e2b351cebd1fd7e1b16f653d (image=quay.io/ceph/ceph:v17.2.0, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mon-a, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, release=754, io.k8s.display-name=CentOS Stream 8, ceph=True, RELEASE=HEAD, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , architecture=x86_64, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., name=centos-stream, distribution-scope=public, vcs-type=git) 2024-04-12T21:49:52.592 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:52 smithi097 podman[24722]: 2024-04-12 21:49:52.340328252 +0000 UTC m=+0.647756194 container cleanup 423ca0f5592d87d69094ccca4b05b2925f9576a7e2b351cebd1fd7e1b16f653d (image=quay.io/ceph/ceph:v17.2.0, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mon-a, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, ceph=True, io.openshift.expose-services=, GIT_BRANCH=HEAD, distribution-scope=public, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, vcs-type=git, version=8) 2024-04-12T21:49:52.592 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:52 smithi097 bash[24722]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mon-a 2024-04-12T21:49:52.592 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:52 smithi097 bash[24751]: Error: no container with name or ID "ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mon.a" found: no such container 2024-04-12T21:49:52.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:52 smithi097 podman[24736]: 2024-04-12 21:49:52.591318211 +0000 UTC m=+0.640516719 container remove 423ca0f5592d87d69094ccca4b05b2925f9576a7e2b351cebd1fd7e1b16f653d (image=quay.io/ceph/ceph:v17.2.0, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mon-a, name=centos-stream, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, RELEASE=HEAD, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, GIT_CLEAN=True, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, vcs-type=git, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, CEPH_POINT_RELEASE=-17.2.0, ceph=True, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, version=8) 2024-04-12T21:49:52.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:52 smithi097 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-12T21:49:52.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:52 smithi097 systemd[1]: Stopped Ceph mon.a for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T21:49:52.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:52 smithi097 systemd[1]: Starting Ceph mon.a for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T21:49:52.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:52 smithi097 podman[24817]: 2024-04-12 21:49:52.808685633 +0000 UTC m=+0.104755732 container create 6e618120f594da25a122688ce6b75f57044039180e09d197734437dbae4e5f7d (image=quay.io/ceph/ceph:v17.2.0, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mon-a, RELEASE=HEAD, GIT_CLEAN=True, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, com.redhat.component=centos-stream-container, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., architecture=x86_64, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, version=8, release=754) 2024-04-12T21:49:52.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:52 smithi097 podman[24817]: 2024-04-12 21:49:52.722166003 +0000 UTC m=+0.018236101 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-04-12T21:49:53.081 INFO:teuthology.orchestra.run.smithi097.stderr:Setting mon public_network to 172.21.0.0/20 2024-04-12T21:49:53.196 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 podman[24817]: 2024-04-12 21:49:53.011640462 +0000 UTC m=+0.307710561 container init 6e618120f594da25a122688ce6b75f57044039180e09d197734437dbae4e5f7d (image=quay.io/ceph/ceph:v17.2.0, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mon-a, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, release=754, RELEASE=HEAD, architecture=x86_64, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, ceph=True, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, name=centos-stream, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870) 2024-04-12T21:49:53.196 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 podman[24817]: 2024-04-12 21:49:53.014490467 +0000 UTC m=+0.310560567 container start 6e618120f594da25a122688ce6b75f57044039180e09d197734437dbae4e5f7d (image=quay.io/ceph/ceph:v17.2.0, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mon-a, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, architecture=x86_64, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, version=8, RELEASE=HEAD, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-12T21:49:53.196 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: set uid:gid to 167:167 (ceph:ceph) 2024-04-12T21:49:53.196 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-04-12T21:49:53.197 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: pidfile_write: ignore empty --pid-file 2024-04-12T21:49:53.197 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: load: jerasure load: lrc 2024-04-12T21:49:53.197 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: RocksDB version: 6.15.5 2024-04-12T21:49:53.197 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-04-12T21:49:53.198 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Compile date Apr 18 2022 2024-04-12T21:49:53.198 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: DB SUMMARY 2024-04-12T21:49:53.198 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: DB Session ID: GS6YEBFV5CA34Y83NUUZ 2024-04-12T21:49:53.198 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: CURRENT file: CURRENT 2024-04-12T21:49:53.198 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: IDENTITY file: IDENTITY 2024-04-12T21:49:53.198 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-04-12T21:49:53.198 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-12T21:49:53.198 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000010.log size: 82637 ; 2024-04-12T21:49:53.198 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.error_if_exists: 0 2024-04-12T21:49:53.198 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.create_if_missing: 0 2024-04-12T21:49:53.198 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.paranoid_checks: 1 2024-04-12T21:49:53.198 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-12T21:49:53.199 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.env: 0x564aaf6c6860 2024-04-12T21:49:53.199 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.fs: Posix File System 2024-04-12T21:49:53.199 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.info_log: 0x564ab149bdc0 2024-04-12T21:49:53.199 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.max_file_opening_threads: 16 2024-04-12T21:49:53.199 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.statistics: (nil) 2024-04-12T21:49:53.199 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.use_fsync: 0 2024-04-12T21:49:53.199 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.max_log_file_size: 0 2024-04-12T21:49:53.199 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-12T21:49:53.199 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-12T21:49:53.199 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.keep_log_file_num: 1000 2024-04-12T21:49:53.199 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.recycle_log_file_num: 0 2024-04-12T21:49:53.199 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.allow_fallocate: 1 2024-04-12T21:49:53.199 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.allow_mmap_reads: 0 2024-04-12T21:49:53.199 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.allow_mmap_writes: 0 2024-04-12T21:49:53.200 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.use_direct_reads: 0 2024-04-12T21:49:53.200 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-12T21:49:53.200 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.create_missing_column_families: 0 2024-04-12T21:49:53.200 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.db_log_dir: 2024-04-12T21:49:53.200 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-04-12T21:49:53.200 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-12T21:49:53.200 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-12T21:49:53.200 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-12T21:49:53.200 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-12T21:49:53.200 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-12T21:49:53.200 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-12T21:49:53.200 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.advise_random_on_open: 1 2024-04-12T21:49:53.200 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.db_write_buffer_size: 0 2024-04-12T21:49:53.200 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.write_buffer_manager: 0x564ab158c2a0 2024-04-12T21:49:53.201 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-12T21:49:53.201 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-04-12T21:49:53.201 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-12T21:49:53.201 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-12T21:49:53.201 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.rate_limiter: (nil) 2024-04-12T21:49:53.201 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-12T21:49:53.201 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.wal_recovery_mode: 2 2024-04-12T21:49:53.201 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.enable_thread_tracking: 0 2024-04-12T21:49:53.201 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.enable_pipelined_write: 0 2024-04-12T21:49:53.201 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.unordered_write: 0 2024-04-12T21:49:53.201 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-12T21:49:53.201 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-12T21:49:53.201 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-12T21:49:53.201 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-12T21:49:53.202 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.row_cache: None 2024-04-12T21:49:53.202 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.wal_filter: None 2024-04-12T21:49:53.202 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-12T21:49:53.202 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.allow_ingest_behind: 0 2024-04-12T21:49:53.202 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.preserve_deletes: 0 2024-04-12T21:49:53.202 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.two_write_queues: 0 2024-04-12T21:49:53.202 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.manual_wal_flush: 0 2024-04-12T21:49:53.202 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.atomic_flush: 0 2024-04-12T21:49:53.202 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-12T21:49:53.202 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-12T21:49:53.202 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-12T21:49:53.202 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.log_readahead_size: 0 2024-04-12T21:49:53.202 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-12T21:49:53.202 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.best_efforts_recovery: 0 2024-04-12T21:49:53.203 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-12T21:49:53.203 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-12T21:49:53.203 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.allow_data_in_errors: 0 2024-04-12T21:49:53.203 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.db_host_id: __hostname__ 2024-04-12T21:49:53.203 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.max_background_jobs: 2 2024-04-12T21:49:53.203 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.max_background_compactions: -1 2024-04-12T21:49:53.203 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.max_subcompactions: 1 2024-04-12T21:49:53.203 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-12T21:49:53.203 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-12T21:49:53.203 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-12T21:49:53.203 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.max_total_wal_size: 0 2024-04-12T21:49:53.203 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-12T21:49:53.203 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-12T21:49:53.203 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-12T21:49:53.204 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-12T21:49:53.204 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.max_open_files: -1 2024-04-12T21:49:53.204 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.bytes_per_sync: 0 2024-04-12T21:49:53.204 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-12T21:49:53.204 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-12T21:49:53.204 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.compaction_readahead_size: 0 2024-04-12T21:49:53.204 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.max_background_flushes: -1 2024-04-12T21:49:53.204 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Compression algorithms supported: 2024-04-12T21:49:53.204 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-12T21:49:53.204 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: kZSTD supported: 0 2024-04-12T21:49:53.204 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: kXpressCompression supported: 0 2024-04-12T21:49:53.204 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: kLZ4HCCompression supported: 1 2024-04-12T21:49:53.204 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: kLZ4Compression supported: 1 2024-04-12T21:49:53.204 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: kBZip2Compression supported: 0 2024-04-12T21:49:53.205 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: kZlibCompression supported: 1 2024-04-12T21:49:53.205 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: kSnappyCompression supported: 1 2024-04-12T21:49:53.205 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-12T21:49:53.205 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2024-04-12T21:49:53.205 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-04-12T21:49:53.205 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-12T21:49:53.205 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.merge_operator: 2024-04-12T21:49:53.205 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.compaction_filter: None 2024-04-12T21:49:53.205 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.compaction_filter_factory: None 2024-04-12T21:49:53.205 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.sst_partitioner_factory: None 2024-04-12T21:49:53.205 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-12T21:49:53.205 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-12T21:49:53.205 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x564ab1467d00) 2024-04-12T21:49:53.205 INFO:journalctl@ceph.mon.a.smithi097.stdout: cache_index_and_filter_blocks: 1 2024-04-12T21:49:53.206 INFO:journalctl@ceph.mon.a.smithi097.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-12T21:49:53.206 INFO:journalctl@ceph.mon.a.smithi097.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-12T21:49:53.206 INFO:journalctl@ceph.mon.a.smithi097.stdout: pin_top_level_index_and_filter: 1 2024-04-12T21:49:53.206 INFO:journalctl@ceph.mon.a.smithi097.stdout: index_type: 0 2024-04-12T21:49:53.206 INFO:journalctl@ceph.mon.a.smithi097.stdout: data_block_index_type: 0 2024-04-12T21:49:53.206 INFO:journalctl@ceph.mon.a.smithi097.stdout: index_shortening: 1 2024-04-12T21:49:53.206 INFO:journalctl@ceph.mon.a.smithi097.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-12T21:49:53.206 INFO:journalctl@ceph.mon.a.smithi097.stdout: hash_index_allow_collision: 1 2024-04-12T21:49:53.206 INFO:journalctl@ceph.mon.a.smithi097.stdout: checksum: 1 2024-04-12T21:49:53.206 INFO:journalctl@ceph.mon.a.smithi097.stdout: no_block_cache: 0 2024-04-12T21:49:53.206 INFO:journalctl@ceph.mon.a.smithi097.stdout: block_cache: 0x564ab14d2170 2024-04-12T21:49:53.206 INFO:journalctl@ceph.mon.a.smithi097.stdout: block_cache_name: BinnedLRUCache 2024-04-12T21:49:53.206 INFO:journalctl@ceph.mon.a.smithi097.stdout: block_cache_options: 2024-04-12T21:49:53.206 INFO:journalctl@ceph.mon.a.smithi097.stdout: capacity : 536870912 2024-04-12T21:49:53.206 INFO:journalctl@ceph.mon.a.smithi097.stdout: num_shard_bits : 4 2024-04-12T21:49:53.207 INFO:journalctl@ceph.mon.a.smithi097.stdout: strict_capacity_limit : 0 2024-04-12T21:49:53.207 INFO:journalctl@ceph.mon.a.smithi097.stdout: high_pri_pool_ratio: 0.000 2024-04-12T21:49:53.207 INFO:journalctl@ceph.mon.a.smithi097.stdout: block_cache_compressed: (nil) 2024-04-12T21:49:53.207 INFO:journalctl@ceph.mon.a.smithi097.stdout: persistent_cache: (nil) 2024-04-12T21:49:53.207 INFO:journalctl@ceph.mon.a.smithi097.stdout: block_size: 4096 2024-04-12T21:49:53.207 INFO:journalctl@ceph.mon.a.smithi097.stdout: block_size_deviation: 10 2024-04-12T21:49:53.207 INFO:journalctl@ceph.mon.a.smithi097.stdout: block_restart_interval: 16 2024-04-12T21:49:53.207 INFO:journalctl@ceph.mon.a.smithi097.stdout: index_block_restart_interval: 1 2024-04-12T21:49:53.207 INFO:journalctl@ceph.mon.a.smithi097.stdout: metadata_block_size: 4096 2024-04-12T21:49:53.207 INFO:journalctl@ceph.mon.a.smithi097.stdout: partition_filters: 0 2024-04-12T21:49:53.207 INFO:journalctl@ceph.mon.a.smithi097.stdout: use_delta_encoding: 1 2024-04-12T21:49:53.207 INFO:journalctl@ceph.mon.a.smithi097.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-04-12T21:49:53.207 INFO:journalctl@ceph.mon.a.smithi097.stdout: whole_key_filtering: 1 2024-04-12T21:49:53.207 INFO:journalctl@ceph.mon.a.smithi097.stdout: verify_compression: 0 2024-04-12T21:49:53.208 INFO:journalctl@ceph.mon.a.smithi097.stdout: read_amp_bytes_per_bit: 0 2024-04-12T21:49:53.208 INFO:journalctl@ceph.mon.a.smithi097.stdout: format_version: 4 2024-04-12T21:49:53.208 INFO:journalctl@ceph.mon.a.smithi097.stdout: enable_index_compression: 1 2024-04-12T21:49:53.208 INFO:journalctl@ceph.mon.a.smithi097.stdout: block_align: 0 2024-04-12T21:49:53.208 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.write_buffer_size: 33554432 2024-04-12T21:49:53.208 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.max_write_buffer_number: 2 2024-04-12T21:49:53.208 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.compression: NoCompression 2024-04-12T21:49:53.208 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.bottommost_compression: Disabled 2024-04-12T21:49:53.208 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.prefix_extractor: nullptr 2024-04-12T21:49:53.208 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-12T21:49:53.208 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.num_levels: 7 2024-04-12T21:49:53.208 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-12T21:49:53.208 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-12T21:49:53.208 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-12T21:49:53.208 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-12T21:49:53.209 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-12T21:49:53.209 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-12T21:49:53.209 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-12T21:49:53.209 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-12T21:49:53.209 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-12T21:49:53.209 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-12T21:49:53.209 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-12T21:49:53.209 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.compression_opts.level: 32767 2024-04-12T21:49:53.209 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.compression_opts.strategy: 0 2024-04-12T21:49:53.209 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-12T21:49:53.209 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-12T21:49:53.209 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-12T21:49:53.209 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.compression_opts.enabled: false 2024-04-12T21:49:53.210 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-12T21:49:53.210 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-12T21:49:53.211 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-12T21:49:53.211 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.target_file_size_base: 67108864 2024-04-12T21:49:53.211 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-12T21:49:53.211 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-12T21:49:53.211 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-12T21:49:53.211 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-12T21:49:53.211 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-12T21:49:53.211 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-12T21:49:53.211 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-12T21:49:53.211 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-12T21:49:53.212 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-12T21:49:53.212 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-12T21:49:53.212 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-12T21:49:53.212 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-12T21:49:53.212 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-12T21:49:53.212 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.arena_block_size: 4194304 2024-04-12T21:49:53.212 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-12T21:49:53.212 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-12T21:49:53.212 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-04-12T21:49:53.212 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.disable_auto_compactions: 0 2024-04-12T21:49:53.212 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-12T21:49:53.212 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-12T21:49:53.212 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-12T21:49:53.213 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-12T21:49:53.213 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-12T21:49:53.213 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-12T21:49:53.213 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-12T21:49:53.213 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-12T21:49:53.213 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-12T21:49:53.213 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-12T21:49:53.213 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.table_properties_collectors: 2024-04-12T21:49:53.213 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.inplace_update_support: 0 2024-04-12T21:49:53.213 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-12T21:49:53.213 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-12T21:49:53.213 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-12T21:49:53.213 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-12T21:49:53.213 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.bloom_locality: 0 2024-04-12T21:49:53.214 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.max_successive_merges: 0 2024-04-12T21:49:53.214 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-12T21:49:53.214 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.paranoid_file_checks: 0 2024-04-12T21:49:53.214 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.force_consistency_checks: 1 2024-04-12T21:49:53.214 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.report_bg_io_stats: 0 2024-04-12T21:49:53.214 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.ttl: 2592000 2024-04-12T21:49:53.214 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-12T21:49:53.214 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.enable_blob_files: false 2024-04-12T21:49:53.214 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.min_blob_size: 0 2024-04-12T21:49:53.214 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.blob_file_size: 268435456 2024-04-12T21:49:53.214 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-12T21:49:53.214 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-12T21:49:53.214 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-12T21:49:53.214 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-12T21:49:53.215 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 5 2024-04-12T21:49:53.215 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: [db/version_set.cc:4083] Creating manifest 13 2024-04-12T21:49:53.215 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712958593040617, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-04-12T21:49:53.215 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #10 mode 2 2024-04-12T21:49:53.215 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-04-12T21:49:53.215 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712958593041587, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 79187, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 77450, "index_size": 225, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10027, "raw_average_key_size": 47, "raw_value_size": 71759, "raw_average_value_size": 340, "num_data_blocks": 10, "num_entries": 211, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1712958593, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "6a0567cf-40f3-4edb-bf08-5bb1d06c76be", "db_session_id": "GS6YEBFV5CA34Y83NUUZ"}} 2024-04-12T21:49:53.215 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: [db/version_set.cc:4083] Creating manifest 15 2024-04-12T21:49:53.215 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712958593042114, "job": 1, "event": "recovery_finished"} 2024-04-12T21:49:53.215 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-12T21:49:53.215 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x564ab14b8a80 2024-04-12T21:49:53.215 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: DB pointer 0x564ab14c8000 2024-04-12T21:49:53.215 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-04-12T21:49:53.215 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-04-12T21:49:53.215 INFO:journalctl@ceph.mon.a.smithi097.stdout: ** DB Stats ** 2024-04-12T21:49:53.216 INFO:journalctl@ceph.mon.a.smithi097.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-12T21:49:53.216 INFO:journalctl@ceph.mon.a.smithi097.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-12T21:49:53.216 INFO:journalctl@ceph.mon.a.smithi097.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-12T21:49:53.216 INFO:journalctl@ceph.mon.a.smithi097.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-12T21:49:53.216 INFO:journalctl@ceph.mon.a.smithi097.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-12T21:49:53.216 INFO:journalctl@ceph.mon.a.smithi097.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-04-12T21:49:53.216 INFO:journalctl@ceph.mon.a.smithi097.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-12T21:49:53.216 INFO:journalctl@ceph.mon.a.smithi097.stdout: 2024-04-12T21:49:53.216 INFO:journalctl@ceph.mon.a.smithi097.stdout: ** Compaction Stats [default] ** 2024-04-12T21:49:53.216 INFO:journalctl@ceph.mon.a.smithi097.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-12T21:49:53.216 INFO:journalctl@ceph.mon.a.smithi097.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-12T21:49:53.216 INFO:journalctl@ceph.mon.a.smithi097.stdout: L0 2/0 79.09 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 117.4 0.00 0.00 1 0.001 0 0 2024-04-12T21:49:53.216 INFO:journalctl@ceph.mon.a.smithi097.stdout: Sum 2/0 79.09 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 117.4 0.00 0.00 1 0.001 0 0 2024-04-12T21:49:53.216 INFO:journalctl@ceph.mon.a.smithi097.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 117.4 0.00 0.00 1 0.001 0 0 2024-04-12T21:49:53.217 INFO:journalctl@ceph.mon.a.smithi097.stdout: 2024-04-12T21:49:53.217 INFO:journalctl@ceph.mon.a.smithi097.stdout: ** Compaction Stats [default] ** 2024-04-12T21:49:53.217 INFO:journalctl@ceph.mon.a.smithi097.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-12T21:49:53.217 INFO:journalctl@ceph.mon.a.smithi097.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-12T21:49:53.217 INFO:journalctl@ceph.mon.a.smithi097.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 117.4 0.00 0.00 1 0.001 0 0 2024-04-12T21:49:53.217 INFO:journalctl@ceph.mon.a.smithi097.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-12T21:49:53.217 INFO:journalctl@ceph.mon.a.smithi097.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-12T21:49:53.217 INFO:journalctl@ceph.mon.a.smithi097.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-12T21:49:53.217 INFO:journalctl@ceph.mon.a.smithi097.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-12T21:49:53.217 INFO:journalctl@ceph.mon.a.smithi097.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-12T21:49:53.217 INFO:journalctl@ceph.mon.a.smithi097.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-12T21:49:53.217 INFO:journalctl@ceph.mon.a.smithi097.stdout: Cumulative compaction: 0.00 GB write, 18.26 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-12T21:49:53.217 INFO:journalctl@ceph.mon.a.smithi097.stdout: Interval compaction: 0.00 GB write, 18.26 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-12T21:49:53.217 INFO:journalctl@ceph.mon.a.smithi097.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-12T21:49:53.218 INFO:journalctl@ceph.mon.a.smithi097.stdout: 2024-04-12T21:49:53.218 INFO:journalctl@ceph.mon.a.smithi097.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-12T21:49:53.218 INFO:journalctl@ceph.mon.a.smithi097.stdout: 2024-04-12T21:49:53.218 INFO:journalctl@ceph.mon.a.smithi097.stdout: ** Compaction Stats [default] ** 2024-04-12T21:49:53.218 INFO:journalctl@ceph.mon.a.smithi097.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-12T21:49:53.218 INFO:journalctl@ceph.mon.a.smithi097.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-12T21:49:53.218 INFO:journalctl@ceph.mon.a.smithi097.stdout: L0 2/0 79.09 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 117.4 0.00 0.00 1 0.001 0 0 2024-04-12T21:49:53.218 INFO:journalctl@ceph.mon.a.smithi097.stdout: Sum 2/0 79.09 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 117.4 0.00 0.00 1 0.001 0 0 2024-04-12T21:49:53.218 INFO:journalctl@ceph.mon.a.smithi097.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-04-12T21:49:53.218 INFO:journalctl@ceph.mon.a.smithi097.stdout: 2024-04-12T21:49:53.218 INFO:journalctl@ceph.mon.a.smithi097.stdout: ** Compaction Stats [default] ** 2024-04-12T21:49:53.218 INFO:journalctl@ceph.mon.a.smithi097.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-12T21:49:53.218 INFO:journalctl@ceph.mon.a.smithi097.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-12T21:49:53.218 INFO:journalctl@ceph.mon.a.smithi097.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 117.4 0.00 0.00 1 0.001 0 0 2024-04-12T21:49:53.219 INFO:journalctl@ceph.mon.a.smithi097.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-12T21:49:53.219 INFO:journalctl@ceph.mon.a.smithi097.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-12T21:49:53.219 INFO:journalctl@ceph.mon.a.smithi097.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-12T21:49:53.219 INFO:journalctl@ceph.mon.a.smithi097.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-12T21:49:53.219 INFO:journalctl@ceph.mon.a.smithi097.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-12T21:49:53.219 INFO:journalctl@ceph.mon.a.smithi097.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-12T21:49:53.219 INFO:journalctl@ceph.mon.a.smithi097.stdout: Cumulative compaction: 0.00 GB write, 18.12 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-12T21:49:53.219 INFO:journalctl@ceph.mon.a.smithi097.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-12T21:49:53.219 INFO:journalctl@ceph.mon.a.smithi097.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-12T21:49:53.219 INFO:journalctl@ceph.mon.a.smithi097.stdout: 2024-04-12T21:49:53.219 INFO:journalctl@ceph.mon.a.smithi097.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-12T21:49:53.219 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: starting mon.a rank 0 at public addrs [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] at bind addrs [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 80bf7680-f916-11ee-bc8d-c7b262605968 2024-04-12T21:49:53.219 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: mon.a@-1(???) e1 preinit fsid 80bf7680-f916-11ee-bc8d-c7b262605968 2024-04-12T21:49:53.219 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: mon.a@-1(???).mds e1 new map 2024-04-12T21:49:53.220 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: mon.a@-1(???).mds e1 print_map 2024-04-12T21:49:53.220 INFO:journalctl@ceph.mon.a.smithi097.stdout: e1 2024-04-12T21:49:53.220 INFO:journalctl@ceph.mon.a.smithi097.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T21:49:53.220 INFO:journalctl@ceph.mon.a.smithi097.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T21:49:53.220 INFO:journalctl@ceph.mon.a.smithi097.stdout: legacy client fscid: -1 2024-04-12T21:49:53.220 INFO:journalctl@ceph.mon.a.smithi097.stdout: 2024-04-12T21:49:53.220 INFO:journalctl@ceph.mon.a.smithi097.stdout: No filesystems configured 2024-04-12T21:49:53.220 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-12T21:49:53.220 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-12T21:49:53.220 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-12T21:49:53.220 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-12T21:49:53.220 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-12T21:49:53.220 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: expand_channel_meta expand map: {default=false} 2024-04-12T21:49:53.220 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: expand_channel_meta from 'false' to 'false' 2024-04-12T21:49:53.221 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: expand_channel_meta expanded map: {default=false} 2024-04-12T21:49:53.221 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: expand_channel_meta expand map: {default=info} 2024-04-12T21:49:53.221 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: expand_channel_meta from 'info' to 'info' 2024-04-12T21:49:53.221 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: expand_channel_meta expanded map: {default=info} 2024-04-12T21:49:53.221 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: expand_channel_meta expand map: {default=daemon} 2024-04-12T21:49:53.221 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: expand_channel_meta from 'daemon' to 'daemon' 2024-04-12T21:49:53.221 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: expand_channel_meta expanded map: {default=daemon} 2024-04-12T21:49:53.221 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: expand_channel_meta expand map: {default=debug} 2024-04-12T21:49:53.221 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: expand_channel_meta from 'debug' to 'debug' 2024-04-12T21:49:53.221 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: expand_channel_meta expanded map: {default=debug} 2024-04-12T21:49:53.221 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 bash[24817]: 6e618120f594da25a122688ce6b75f57044039180e09d197734437dbae4e5f7d 2024-04-12T21:49:53.221 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 systemd[1]: Started Ceph mon.a for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T21:49:53.221 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-12T21:49:53.221 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: monmap e1: 1 mons at {a=[v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0]} 2024-04-12T21:49:53.222 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: fsmap 2024-04-12T21:49:53.222 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: osdmap e1: 0 total, 0 up, 0 in 2024-04-12T21:49:53.222 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:53 smithi097 ceph-mon[24835]: mgrmap e1: no daemons active 2024-04-12T21:49:54.179 INFO:teuthology.orchestra.run.smithi097.stderr:Wrote config to /etc/ceph/ceph.conf 2024-04-12T21:49:54.180 INFO:teuthology.orchestra.run.smithi097.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-12T21:49:54.180 INFO:teuthology.orchestra.run.smithi097.stderr:Creating mgr... 2024-04-12T21:49:54.180 INFO:teuthology.orchestra.run.smithi097.stderr:Verifying port 9283 ... 2024-04-12T21:49:54.351 INFO:teuthology.orchestra.run.smithi097.stderr:systemctl: Failed to reset failed state of unit ceph-80bf7680-f916-11ee-bc8d-c7b262605968@mgr.y.service: Unit ceph-80bf7680-f916-11ee-bc8d-c7b262605968@mgr.y.service not loaded. 2024-04-12T21:49:54.358 INFO:teuthology.orchestra.run.smithi097.stderr:systemctl: Created symlink /etc/systemd/system/ceph-80bf7680-f916-11ee-bc8d-c7b262605968.target.wants/ceph-80bf7680-f916-11ee-bc8d-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-80bf7680-f916-11ee-bc8d-c7b262605968@.service. 2024-04-12T21:49:54.818 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:54 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/3432609240' entity='client.admin' 2024-04-12T21:49:55.003 INFO:teuthology.orchestra.run.smithi097.stderr:systemctl: disabled 2024-04-12T21:49:55.010 INFO:teuthology.orchestra.run.smithi097.stderr:systemctl: inactive 2024-04-12T21:49:55.010 INFO:teuthology.orchestra.run.smithi097.stderr:firewalld.service is not enabled 2024-04-12T21:49:55.010 INFO:teuthology.orchestra.run.smithi097.stderr:Not possible to enable service . firewalld.service is not available 2024-04-12T21:49:55.018 INFO:teuthology.orchestra.run.smithi097.stderr:systemctl: disabled 2024-04-12T21:49:55.025 INFO:teuthology.orchestra.run.smithi097.stderr:systemctl: inactive 2024-04-12T21:49:55.026 INFO:teuthology.orchestra.run.smithi097.stderr:firewalld.service is not enabled 2024-04-12T21:49:55.026 INFO:teuthology.orchestra.run.smithi097.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-04-12T21:49:55.026 INFO:teuthology.orchestra.run.smithi097.stderr:Waiting for mgr to start... 2024-04-12T21:49:55.026 INFO:teuthology.orchestra.run.smithi097.stderr:Waiting for mgr... 2024-04-12T21:49:55.151 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:49:54 smithi097 systemd[1]: Started Ceph mgr.y for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T21:49:55.410 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:49:55 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:49:55.396+0000 7f69b6589000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-12T21:49:55.517 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: 2024-04-12T21:49:55.517 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: { 2024-04-12T21:49:55.517 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "fsid": "80bf7680-f916-11ee-bc8d-c7b262605968", 2024-04-12T21:49:55.518 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "health": { 2024-04-12T21:49:55.518 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-12T21:49:55.518 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "checks": {}, 2024-04-12T21:49:55.518 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "mutes": [] 2024-04-12T21:49:55.518 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: }, 2024-04-12T21:49:55.518 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-12T21:49:55.518 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "quorum": [ 2024-04-12T21:49:55.518 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: 0 2024-04-12T21:49:55.518 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: ], 2024-04-12T21:49:55.518 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-12T21:49:55.519 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "a" 2024-04-12T21:49:55.519 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: ], 2024-04-12T21:49:55.519 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-04-12T21:49:55.519 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "monmap": { 2024-04-12T21:49:55.519 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-12T21:49:55.519 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-12T21:49:55.519 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-12T21:49:55.519 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: }, 2024-04-12T21:49:55.519 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "osdmap": { 2024-04-12T21:49:55.519 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-12T21:49:55.519 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-12T21:49:55.519 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-12T21:49:55.519 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-12T21:49:55.519 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-12T21:49:55.524 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-12T21:49:55.524 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-12T21:49:55.524 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: }, 2024-04-12T21:49:55.524 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "pgmap": { 2024-04-12T21:49:55.524 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-12T21:49:55.524 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-12T21:49:55.525 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-12T21:49:55.525 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-12T21:49:55.525 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-12T21:49:55.525 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-12T21:49:55.525 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-12T21:49:55.525 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-12T21:49:55.525 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: }, 2024-04-12T21:49:55.525 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "fsmap": { 2024-04-12T21:49:55.526 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-12T21:49:55.526 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-12T21:49:55.526 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-12T21:49:55.526 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: }, 2024-04-12T21:49:55.526 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-12T21:49:55.526 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "available": false, 2024-04-12T21:49:55.526 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-12T21:49:55.527 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "modules": [ 2024-04-12T21:49:55.527 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "iostat", 2024-04-12T21:49:55.527 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "nfs", 2024-04-12T21:49:55.527 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "restful" 2024-04-12T21:49:55.527 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: ], 2024-04-12T21:49:55.527 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "services": {} 2024-04-12T21:49:55.527 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: }, 2024-04-12T21:49:55.527 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "servicemap": { 2024-04-12T21:49:55.528 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-12T21:49:55.528 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "modified": "2024-04-12T21:49:48.427445+0000", 2024-04-12T21:49:55.528 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "services": {} 2024-04-12T21:49:55.528 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: }, 2024-04-12T21:49:55.528 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-12T21:49:55.528 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: } 2024-04-12T21:49:55.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:55 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/277527478' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-12T21:49:55.911 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:49:55 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:49:55.503+0000 7f69b6589000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-12T21:49:55.977 INFO:teuthology.orchestra.run.smithi097.stderr:mgr not available, waiting (1/15)... 2024-04-12T21:49:56.271 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:49:56 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:49:56.102+0000 7f69b6589000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-12T21:49:56.271 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:49:56 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:49:56.188+0000 7f69b6589000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-12T21:49:56.271 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:49:56 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:49:56.269+0000 7f69b6589000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-12T21:49:56.601 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:49:56 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:49:56.415+0000 7f69b6589000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-12T21:49:56.601 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:49:56 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:49:56.493+0000 7f69b6589000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-12T21:49:56.601 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:49:56 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:49:56.599+0000 7f69b6589000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-12T21:49:56.910 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:49:56 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:49:56.681+0000 7f69b6589000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-12T21:49:56.911 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:49:56 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:49:56.840+0000 7f69b6589000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-12T21:49:57.410 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:49:56 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:49:56.922+0000 7f69b6589000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-12T21:49:58.251 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:49:57 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:49:57.934+0000 7f69b6589000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-12T21:49:58.251 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:49:58 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:49:58.068+0000 7f69b6589000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-12T21:49:58.531 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:49:58 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:49:58.256+0000 7f69b6589000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-12T21:49:58.532 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:49:58 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:49:58.342+0000 7f69b6589000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-12T21:49:58.532 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:49:58 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:49:58.480+0000 7f69b6589000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-12T21:49:58.538 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: 2024-04-12T21:49:58.538 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: { 2024-04-12T21:49:58.538 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "fsid": "80bf7680-f916-11ee-bc8d-c7b262605968", 2024-04-12T21:49:58.538 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "health": { 2024-04-12T21:49:58.538 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-12T21:49:58.538 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "checks": {}, 2024-04-12T21:49:58.538 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "mutes": [] 2024-04-12T21:49:58.538 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: }, 2024-04-12T21:49:58.538 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-12T21:49:58.538 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "quorum": [ 2024-04-12T21:49:58.538 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: 0 2024-04-12T21:49:58.538 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: ], 2024-04-12T21:49:58.539 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-12T21:49:58.539 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "a" 2024-04-12T21:49:58.539 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: ], 2024-04-12T21:49:58.539 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "quorum_age": 5, 2024-04-12T21:49:58.539 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "monmap": { 2024-04-12T21:49:58.539 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-12T21:49:58.539 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-12T21:49:58.539 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-12T21:49:58.539 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: }, 2024-04-12T21:49:58.539 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "osdmap": { 2024-04-12T21:49:58.539 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-12T21:49:58.539 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-12T21:49:58.539 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-12T21:49:58.539 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-12T21:49:58.539 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-12T21:49:58.540 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-12T21:49:58.540 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-12T21:49:58.540 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: }, 2024-04-12T21:49:58.540 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "pgmap": { 2024-04-12T21:49:58.540 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-12T21:49:58.540 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-12T21:49:58.540 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-12T21:49:58.540 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-12T21:49:58.540 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-12T21:49:58.540 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-12T21:49:58.540 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-12T21:49:58.540 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-12T21:49:58.541 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: }, 2024-04-12T21:49:58.541 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "fsmap": { 2024-04-12T21:49:58.541 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-12T21:49:58.541 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-12T21:49:58.541 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-12T21:49:58.541 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: }, 2024-04-12T21:49:58.541 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-12T21:49:58.541 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "available": false, 2024-04-12T21:49:58.541 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-12T21:49:58.541 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "modules": [ 2024-04-12T21:49:58.541 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "iostat", 2024-04-12T21:49:58.541 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "nfs", 2024-04-12T21:49:58.541 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "restful" 2024-04-12T21:49:58.541 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: ], 2024-04-12T21:49:58.541 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "services": {} 2024-04-12T21:49:58.542 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: }, 2024-04-12T21:49:58.542 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "servicemap": { 2024-04-12T21:49:58.542 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-12T21:49:58.542 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "modified": "2024-04-12T21:49:48.427445+0000", 2024-04-12T21:49:58.542 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "services": {} 2024-04-12T21:49:58.542 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: }, 2024-04-12T21:49:58.542 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-12T21:49:58.542 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: } 2024-04-12T21:49:59.117 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:49:59 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/1830555443' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-12T21:49:59.117 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:49:58 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:49:58.813+0000 7f69b6589000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-12T21:49:59.117 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:49:59 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:49:59.031+0000 7f69b6589000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-12T21:49:59.117 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:49:59 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:49:59.115+0000 7f69b6589000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-12T21:49:59.215 INFO:teuthology.orchestra.run.smithi097.stderr:mgr not available, waiting (2/15)... 2024-04-12T21:49:59.410 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:49:59 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:49:59.287+0000 7f69b6589000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-12T21:50:00.474 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:00 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:50:00.187+0000 7f69b6589000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-12T21:50:00.766 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:00 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:50:00.473+0000 7f69b6589000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-12T21:50:00.766 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:00 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:50:00.565+0000 7f69b6589000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-12T21:50:00.766 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:00 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:50:00.712+0000 7f69b6589000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-12T21:50:01.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:00 smithi097 ceph-mon[24835]: Activating manager daemon y 2024-04-12T21:50:01.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:00 smithi097 ceph-mon[24835]: mgrmap e2: y(active, starting, since 0.00251886s) 2024-04-12T21:50:01.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:00 smithi097 ceph-mon[24835]: from='mgr.14100 172.21.15.97:0/1058972667' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-12T21:50:01.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:00 smithi097 ceph-mon[24835]: from='mgr.14100 172.21.15.97:0/1058972667' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-12T21:50:01.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:00 smithi097 ceph-mon[24835]: from='mgr.14100 172.21.15.97:0/1058972667' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-12T21:50:01.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:00 smithi097 ceph-mon[24835]: from='mgr.14100 172.21.15.97:0/1058972667' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-12T21:50:01.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:00 smithi097 ceph-mon[24835]: from='mgr.14100 172.21.15.97:0/1058972667' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-12T21:50:01.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:00 smithi097 ceph-mon[24835]: Manager daemon y is now available 2024-04-12T21:50:01.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:00 smithi097 ceph-mon[24835]: from='mgr.14100 172.21.15.97:0/1058972667' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T21:50:01.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:00 smithi097 ceph-mon[24835]: from='mgr.14100 172.21.15.97:0/1058972667' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T21:50:01.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:00 smithi097 ceph-mon[24835]: from='mgr.14100 172.21.15.97:0/1058972667' entity='mgr.y' 2024-04-12T21:50:01.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:00 smithi097 ceph-mon[24835]: from='mgr.14100 172.21.15.97:0/1058972667' entity='mgr.y' 2024-04-12T21:50:01.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:00 smithi097 ceph-mon[24835]: from='mgr.14100 172.21.15.97:0/1058972667' entity='mgr.y' 2024-04-12T21:50:01.710 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: 2024-04-12T21:50:01.711 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: { 2024-04-12T21:50:01.711 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "fsid": "80bf7680-f916-11ee-bc8d-c7b262605968", 2024-04-12T21:50:01.711 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "health": { 2024-04-12T21:50:01.711 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-12T21:50:01.711 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "checks": {}, 2024-04-12T21:50:01.711 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "mutes": [] 2024-04-12T21:50:01.711 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: }, 2024-04-12T21:50:01.711 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-12T21:50:01.711 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "quorum": [ 2024-04-12T21:50:01.711 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: 0 2024-04-12T21:50:01.711 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: ], 2024-04-12T21:50:01.711 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-12T21:50:01.711 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "a" 2024-04-12T21:50:01.711 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: ], 2024-04-12T21:50:01.712 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "quorum_age": 8, 2024-04-12T21:50:01.712 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "monmap": { 2024-04-12T21:50:01.712 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-12T21:50:01.712 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-12T21:50:01.712 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-12T21:50:01.712 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: }, 2024-04-12T21:50:01.712 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "osdmap": { 2024-04-12T21:50:01.712 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-12T21:50:01.712 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-12T21:50:01.712 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-12T21:50:01.712 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-12T21:50:01.712 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-12T21:50:01.716 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-12T21:50:01.716 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-12T21:50:01.716 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: }, 2024-04-12T21:50:01.716 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "pgmap": { 2024-04-12T21:50:01.716 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-12T21:50:01.716 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-12T21:50:01.716 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-12T21:50:01.716 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-12T21:50:01.716 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-12T21:50:01.716 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-12T21:50:01.717 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-12T21:50:01.717 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-12T21:50:01.717 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: }, 2024-04-12T21:50:01.717 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "fsmap": { 2024-04-12T21:50:01.717 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-12T21:50:01.717 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-12T21:50:01.717 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-12T21:50:01.717 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: }, 2024-04-12T21:50:01.717 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-12T21:50:01.717 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "available": false, 2024-04-12T21:50:01.717 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-12T21:50:01.717 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "modules": [ 2024-04-12T21:50:01.717 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "iostat", 2024-04-12T21:50:01.717 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "nfs", 2024-04-12T21:50:01.717 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "restful" 2024-04-12T21:50:01.718 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: ], 2024-04-12T21:50:01.718 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "services": {} 2024-04-12T21:50:01.718 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: }, 2024-04-12T21:50:01.718 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "servicemap": { 2024-04-12T21:50:01.718 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-12T21:50:01.718 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "modified": "2024-04-12T21:49:48.427445+0000", 2024-04-12T21:50:01.718 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "services": {} 2024-04-12T21:50:01.718 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: }, 2024-04-12T21:50:01.718 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-12T21:50:01.718 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: } 2024-04-12T21:50:01.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:01 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/521402901' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-12T21:50:01.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:01 smithi097 ceph-mon[24835]: mgrmap e3: y(active, since 1.00582s) 2024-04-12T21:50:02.254 INFO:teuthology.orchestra.run.smithi097.stderr:mgr not available, waiting (3/15)... 2024-04-12T21:50:04.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:03 smithi097 ceph-mon[24835]: mgrmap e4: y(active, since 2s) 2024-04-12T21:50:08.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:07 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/1756355591' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-12T21:50:08.722 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: 2024-04-12T21:50:08.722 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: { 2024-04-12T21:50:08.722 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "fsid": "80bf7680-f916-11ee-bc8d-c7b262605968", 2024-04-12T21:50:08.722 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "health": { 2024-04-12T21:50:08.722 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-12T21:50:08.723 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "checks": {}, 2024-04-12T21:50:08.723 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "mutes": [] 2024-04-12T21:50:08.723 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: }, 2024-04-12T21:50:08.723 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-12T21:50:08.723 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "quorum": [ 2024-04-12T21:50:08.723 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: 0 2024-04-12T21:50:08.723 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: ], 2024-04-12T21:50:08.723 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-12T21:50:08.723 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "a" 2024-04-12T21:50:08.723 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: ], 2024-04-12T21:50:08.723 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "quorum_age": 14, 2024-04-12T21:50:08.723 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "monmap": { 2024-04-12T21:50:08.723 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-12T21:50:08.724 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-12T21:50:08.724 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-12T21:50:08.724 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: }, 2024-04-12T21:50:08.724 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "osdmap": { 2024-04-12T21:50:08.724 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-12T21:50:08.724 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-12T21:50:08.724 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-12T21:50:08.724 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-12T21:50:08.724 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-12T21:50:08.728 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-12T21:50:08.728 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-12T21:50:08.728 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: }, 2024-04-12T21:50:08.728 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "pgmap": { 2024-04-12T21:50:08.728 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-12T21:50:08.728 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-12T21:50:08.728 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-12T21:50:08.729 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-12T21:50:08.729 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-12T21:50:08.729 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-12T21:50:08.729 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-12T21:50:08.729 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-12T21:50:08.729 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: }, 2024-04-12T21:50:08.729 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "fsmap": { 2024-04-12T21:50:08.729 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-12T21:50:08.729 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-12T21:50:08.729 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-12T21:50:08.729 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: }, 2024-04-12T21:50:08.729 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-12T21:50:08.729 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "available": true, 2024-04-12T21:50:08.729 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-12T21:50:08.730 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "modules": [ 2024-04-12T21:50:08.730 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "iostat", 2024-04-12T21:50:08.730 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "nfs", 2024-04-12T21:50:08.730 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "restful" 2024-04-12T21:50:08.730 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: ], 2024-04-12T21:50:08.730 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "services": {} 2024-04-12T21:50:08.730 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: }, 2024-04-12T21:50:08.730 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "servicemap": { 2024-04-12T21:50:08.730 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-12T21:50:08.730 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "modified": "2024-04-12T21:49:48.427445+0000", 2024-04-12T21:50:08.730 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "services": {} 2024-04-12T21:50:08.730 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: }, 2024-04-12T21:50:08.730 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-12T21:50:08.730 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: } 2024-04-12T21:50:12.954 INFO:teuthology.orchestra.run.smithi097.stderr:mgr is available 2024-04-12T21:50:13.536 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: 2024-04-12T21:50:13.536 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: [global] 2024-04-12T21:50:13.536 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: fsid = 80bf7680-f916-11ee-bc8d-c7b262605968 2024-04-12T21:50:13.536 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-04-12T21:50:13.536 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-04-12T21:50:13.536 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-04-12T21:50:13.536 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-04-12T21:50:13.536 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: 2024-04-12T21:50:13.537 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: [mgr] 2024-04-12T21:50:13.537 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-04-12T21:50:13.537 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: 2024-04-12T21:50:13.537 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: [osd] 2024-04-12T21:50:13.537 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-04-12T21:50:13.537 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-04-12T21:50:14.084 INFO:teuthology.orchestra.run.smithi097.stderr:Enabling cephadm module... 2024-04-12T21:50:14.358 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:14 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/2032696758' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-12T21:50:14.359 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:14 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/2032696758' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-12T21:50:15.364 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:15 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/796552254' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-12T21:50:15.364 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:15 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: ignoring --setuser ceph since I am not root 2024-04-12T21:50:15.364 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:15 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: ignoring --setgroup ceph since I am not root 2024-04-12T21:50:15.660 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:15 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:50:15.548+0000 7fb55abe2000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-12T21:50:16.056 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:15 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:50:15.635+0000 7fb55abe2000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-12T21:50:16.317 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:16 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:50:16.226+0000 7fb55abe2000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-12T21:50:16.317 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:16 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/796552254' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-12T21:50:16.318 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:16 smithi097 ceph-mon[24835]: mgrmap e5: y(active, since 14s) 2024-04-12T21:50:16.635 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:16 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:50:16.315+0000 7fb55abe2000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-12T21:50:16.635 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:16 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:50:16.401+0000 7fb55abe2000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-12T21:50:16.635 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:16 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:50:16.550+0000 7fb55abe2000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-12T21:50:16.635 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:16 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:50:16.633+0000 7fb55abe2000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-12T21:50:16.808 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: { 2024-04-12T21:50:16.808 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "epoch": 5, 2024-04-12T21:50:16.808 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "available": true, 2024-04-12T21:50:16.808 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "active_name": "y", 2024-04-12T21:50:16.808 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_standby": 0 2024-04-12T21:50:16.808 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: } 2024-04-12T21:50:16.910 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:16 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:50:16.743+0000 7fb55abe2000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-12T21:50:16.910 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:16 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:50:16.829+0000 7fb55abe2000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-12T21:50:17.283 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:17 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/547697854' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-12T21:50:17.283 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:16 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:50:16.994+0000 7fb55abe2000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-12T21:50:17.283 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:17 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:50:17.078+0000 7fb55abe2000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-12T21:50:17.447 INFO:teuthology.orchestra.run.smithi097.stderr:Waiting for the mgr to restart... 2024-04-12T21:50:17.447 INFO:teuthology.orchestra.run.smithi097.stderr:Waiting for mgr epoch 5... 2024-04-12T21:50:18.410 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:18 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:50:18.105+0000 7fb55abe2000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-12T21:50:18.411 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:18 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:50:18.232+0000 7fb55abe2000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-12T21:50:18.911 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:18 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:50:18.418+0000 7fb55abe2000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-12T21:50:18.911 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:18 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:50:18.504+0000 7fb55abe2000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-12T21:50:18.911 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:18 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:50:18.636+0000 7fb55abe2000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-12T21:50:19.251 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:18 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:50:18.948+0000 7fb55abe2000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-12T21:50:19.251 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:19 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:50:19.166+0000 7fb55abe2000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-12T21:50:19.251 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:19 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:50:19.250+0000 7fb55abe2000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-12T21:50:19.660 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:19 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:50:19.416+0000 7fb55abe2000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-12T21:50:20.628 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:20 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:50:20.329+0000 7fb55abe2000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-12T21:50:20.628 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:20 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:50:20.626+0000 7fb55abe2000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-12T21:50:20.910 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:20 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:50:20.718+0000 7fb55abe2000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-12T21:50:20.911 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:20 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:50:20.868+0000 7fb55abe2000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-12T21:50:21.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:20 smithi097 ceph-mon[24835]: Active manager daemon y restarted 2024-04-12T21:50:21.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:20 smithi097 ceph-mon[24835]: Activating manager daemon y 2024-04-12T21:50:21.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:20 smithi097 ceph-mon[24835]: osdmap e2: 0 total, 0 up, 0 in 2024-04-12T21:50:21.937 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: { 2024-04-12T21:50:21.937 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-04-12T21:50:21.937 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "initialized": true 2024-04-12T21:50:21.937 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: } 2024-04-12T21:50:22.267 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:21 smithi097 ceph-mon[24835]: mgrmap e6: y(active, starting, since 0.0523581s) 2024-04-12T21:50:22.267 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:21 smithi097 ceph-mon[24835]: from='mgr.14120 172.21.15.97:0/1782092263' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-12T21:50:22.267 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:21 smithi097 ceph-mon[24835]: from='mgr.14120 172.21.15.97:0/1782092263' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-12T21:50:22.267 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:21 smithi097 ceph-mon[24835]: from='mgr.14120 172.21.15.97:0/1782092263' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-12T21:50:22.267 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:21 smithi097 ceph-mon[24835]: from='mgr.14120 172.21.15.97:0/1782092263' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-12T21:50:22.267 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:21 smithi097 ceph-mon[24835]: from='mgr.14120 172.21.15.97:0/1782092263' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-12T21:50:22.267 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:21 smithi097 ceph-mon[24835]: Manager daemon y is now available 2024-04-12T21:50:22.267 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:21 smithi097 ceph-mon[24835]: from='mgr.14120 172.21.15.97:0/1782092263' entity='mgr.y' 2024-04-12T21:50:22.267 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:21 smithi097 ceph-mon[24835]: from='mgr.14120 172.21.15.97:0/1782092263' entity='mgr.y' 2024-04-12T21:50:22.267 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:21 smithi097 ceph-mon[24835]: from='mgr.14120 172.21.15.97:0/1782092263' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:50:22.267 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:21 smithi097 ceph-mon[24835]: from='mgr.14120 172.21.15.97:0/1782092263' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:50:22.267 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:21 smithi097 ceph-mon[24835]: from='mgr.14120 172.21.15.97:0/1782092263' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:50:22.268 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:21 smithi097 ceph-mon[24835]: from='mgr.14120 172.21.15.97:0/1782092263' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T21:50:22.268 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:21 smithi097 ceph-mon[24835]: from='mgr.14120 172.21.15.97:0/1782092263' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T21:50:22.559 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:22 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: [12/Apr/2024:21:50:22] ENGINE Bus STARTING 2024-04-12T21:50:22.559 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:22 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: [12/Apr/2024:21:50:22] ENGINE Serving on https://172.21.15.97:7150 2024-04-12T21:50:22.559 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:22 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: [12/Apr/2024:21:50:22] ENGINE Bus STARTED 2024-04-12T21:50:22.562 INFO:teuthology.orchestra.run.smithi097.stderr:mgr epoch 5 is available 2024-04-12T21:50:22.562 INFO:teuthology.orchestra.run.smithi097.stderr:Setting orchestrator backend to cephadm... 2024-04-12T21:50:23.142 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:22 smithi097 ceph-mon[24835]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-12T21:50:23.143 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:22 smithi097 ceph-mon[24835]: mgrmap e7: y(active, since 1.05916s) 2024-04-12T21:50:23.143 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:22 smithi097 ceph-mon[24835]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-12T21:50:23.143 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:22 smithi097 ceph-mon[24835]: from='mgr.14120 172.21.15.97:0/1782092263' entity='mgr.y' 2024-04-12T21:50:23.143 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:22 smithi097 ceph-mon[24835]: from='mgr.14120 172.21.15.97:0/1782092263' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:50:24.208 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: value unchanged 2024-04-12T21:50:24.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:24 smithi097 ceph-mon[24835]: [12/Apr/2024:21:50:22] ENGINE Bus STARTING 2024-04-12T21:50:24.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:24 smithi097 ceph-mon[24835]: [12/Apr/2024:21:50:22] ENGINE Serving on https://172.21.15.97:7150 2024-04-12T21:50:24.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:24 smithi097 ceph-mon[24835]: [12/Apr/2024:21:50:22] ENGINE Bus STARTED 2024-04-12T21:50:24.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:24 smithi097 ceph-mon[24835]: from='mgr.14120 172.21.15.97:0/1782092263' entity='mgr.y' 2024-04-12T21:50:24.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:24 smithi097 ceph-mon[24835]: from='mgr.14120 172.21.15.97:0/1782092263' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:50:24.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:24 smithi097 ceph-mon[24835]: mgrmap e8: y(active, since 2s) 2024-04-12T21:50:24.782 INFO:teuthology.orchestra.run.smithi097.stderr:Generating ssh key... 2024-04-12T21:50:25.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:25 smithi097 ceph-mon[24835]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:50:25.911 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:25 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: Generating public/private rsa key pair. 2024-04-12T21:50:25.911 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:25 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: Your identification has been saved in /tmp/tmpsf9fyivl/key. 2024-04-12T21:50:25.911 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:25 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: Your public key has been saved in /tmp/tmpsf9fyivl/key.pub. 2024-04-12T21:50:25.911 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:25 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: The key fingerprint is: 2024-04-12T21:50:25.911 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:25 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: SHA256:QBKhOAEcp15ZNNUzMYYQxYu0mHFaMa3hl2TU2fgtz0A ceph-80bf7680-f916-11ee-bc8d-c7b262605968 2024-04-12T21:50:25.911 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:25 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: The key's randomart image is: 2024-04-12T21:50:25.911 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:25 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: +---[RSA 3072]----+ 2024-04-12T21:50:25.911 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:25 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: |=...=@@+++= | 2024-04-12T21:50:25.911 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:25 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: | +oooBo*.B.E | 2024-04-12T21:50:25.911 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:25 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: |o..oX O o = . | 2024-04-12T21:50:25.911 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:25 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: |...+ = = + . | 2024-04-12T21:50:25.912 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:25 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: | . . S = | 2024-04-12T21:50:25.912 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:25 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: | o | 2024-04-12T21:50:25.912 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:25 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: | | 2024-04-12T21:50:25.912 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:25 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: | | 2024-04-12T21:50:25.912 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:25 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: | | 2024-04-12T21:50:25.912 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:25 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: +----[SHA256]-----+ 2024-04-12T21:50:26.212 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:26 smithi097 ceph-mon[24835]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:50:26.212 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:26 smithi097 ceph-mon[24835]: from='mgr.14120 172.21.15.97:0/1782092263' entity='mgr.y' 2024-04-12T21:50:26.212 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:26 smithi097 ceph-mon[24835]: from='mgr.14120 172.21.15.97:0/1782092263' entity='mgr.y' 2024-04-12T21:50:26.813 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCuO/R8dfmju8cc46G8LpiLuesKcxRF24sV0lpnRQ7vYOUop74Btl4FF1Khi8iWfUJ3GG2YlxkAOc76SNs3f+ko6u5mMcNzAkerP6vPD7GFO4PNBVrOmc6CeWmSycHZAoLkppLcMIFz6fpXgMu2ekqrGqozWjUgVY0C7Zc4kEUojm9z3yxGaCQYb8Mcm0znqRK1pjfHRjjLrnEG1t/lWA0odg0RwxQm0imuTCitlekFp+d8kNu6BC0XhGrro65J9zmrVWgCfpkE0aUqwRauSvaE04actKG8Hml3/Buu4q5p2XU45rsLS3GTcZ4dZVPm+nj1IvOCaqv0tBA6XUEWuDq20m5GKPdd7tla2CUh2iW34cfwBwjMRso+HhpQDXLQLNJSvCHIG6sOTpPimANJjC22xsnd1kCXkOU/cKxBiFJQS697kT09BfEuu9pWSRAk7l5GZjay4Q+BO0AvG/aeuowjbtkG3FFlHrBESTcvMZkut3W894H51ckMi6mv6nHczGE= ceph-80bf7680-f916-11ee-bc8d-c7b262605968 2024-04-12T21:50:27.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:27 smithi097 ceph-mon[24835]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:50:27.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:27 smithi097 ceph-mon[24835]: Generating ssh key... 2024-04-12T21:50:27.371 INFO:teuthology.orchestra.run.smithi097.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-12T21:50:27.371 INFO:teuthology.orchestra.run.smithi097.stderr:Adding key to root@localhost authorized_keys... 2024-04-12T21:50:27.371 INFO:teuthology.orchestra.run.smithi097.stderr:Adding host smithi097... 2024-04-12T21:50:28.303 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:28 smithi097 ceph-mon[24835]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:50:28.980 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: Added host 'smithi097' with addr '172.21.15.97' 2024-04-12T21:50:29.309 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:29 smithi097 ceph-mon[24835]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi097", "addr": "172.21.15.97", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:50:29.310 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:29 smithi097 ceph-mon[24835]: from='mgr.14120 172.21.15.97:0/1782092263' entity='mgr.y' 2024-04-12T21:50:29.310 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:29 smithi097 ceph-mon[24835]: from='mgr.14120 172.21.15.97:0/1782092263' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:50:29.579 INFO:teuthology.orchestra.run.smithi097.stderr:Deploying unmanaged mon service... 2024-04-12T21:50:30.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:30 smithi097 ceph-mon[24835]: Deploying cephadm binary to smithi097 2024-04-12T21:50:30.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:30 smithi097 ceph-mon[24835]: Added host smithi097 2024-04-12T21:50:30.219 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: Scheduled mon update... 2024-04-12T21:50:30.861 INFO:teuthology.orchestra.run.smithi097.stderr:Deploying unmanaged mgr service... 2024-04-12T21:50:31.472 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:31 smithi097 ceph-mon[24835]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:50:31.472 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:31 smithi097 ceph-mon[24835]: Saving service mon spec with placement count:5 2024-04-12T21:50:31.473 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:31 smithi097 ceph-mon[24835]: from='mgr.14120 172.21.15.97:0/1782092263' entity='mgr.y' 2024-04-12T21:50:31.702 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-04-12T21:50:32.891 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:32 smithi097 ceph-mon[24835]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:50:32.891 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:32 smithi097 ceph-mon[24835]: Saving service mgr spec with placement count:2 2024-04-12T21:50:32.891 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:32 smithi097 ceph-mon[24835]: from='mgr.14120 172.21.15.97:0/1782092263' entity='mgr.y' 2024-04-12T21:50:34.156 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:33 smithi097 ceph-mon[24835]: from='mgr.14120 172.21.15.97:0/1782092263' entity='mgr.y' 2024-04-12T21:50:34.156 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:33 smithi097 ceph-mon[24835]: from='mgr.14120 172.21.15.97:0/1782092263' entity='mgr.y' 2024-04-12T21:50:34.157 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:33 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/2671570437' entity='client.admin' 2024-04-12T21:50:35.354 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:35 smithi097 ceph-mon[24835]: from='mgr.14120 172.21.15.97:0/1782092263' entity='mgr.y' 2024-04-12T21:50:35.903 INFO:teuthology.orchestra.run.smithi097.stderr:Enabling the dashboard module... 2024-04-12T21:50:36.592 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:36 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/3886350682' entity='client.admin' 2024-04-12T21:50:36.593 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:36 smithi097 ceph-mon[24835]: from='mgr.14120 172.21.15.97:0/1782092263' entity='mgr.y' 2024-04-12T21:50:37.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:37 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/707227674' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-12T21:50:37.661 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:37 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: ignoring --setuser ceph since I am not root 2024-04-12T21:50:37.661 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:37 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: ignoring --setgroup ceph since I am not root 2024-04-12T21:50:37.925 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:37 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:50:37.836+0000 7fa7bbc13000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-12T21:50:37.926 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:37 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:50:37.924+0000 7fa7bbc13000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-12T21:50:38.531 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:38 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/707227674' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-12T21:50:38.532 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:38 smithi097 ceph-mon[24835]: mgrmap e9: y(active, since 16s) 2024-04-12T21:50:38.788 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:38 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:50:38.529+0000 7fa7bbc13000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-12T21:50:38.788 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:38 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:50:38.622+0000 7fa7bbc13000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-12T21:50:38.788 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:38 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:50:38.706+0000 7fa7bbc13000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-12T21:50:38.789 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: { 2024-04-12T21:50:38.789 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "epoch": 9, 2024-04-12T21:50:38.789 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "available": true, 2024-04-12T21:50:38.789 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "active_name": "y", 2024-04-12T21:50:38.790 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "num_standby": 0 2024-04-12T21:50:38.790 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: } 2024-04-12T21:50:39.056 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:38 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:50:38.864+0000 7fa7bbc13000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-12T21:50:39.056 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:38 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:50:38.945+0000 7fa7bbc13000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-12T21:50:39.056 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:39 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:50:39.054+0000 7fa7bbc13000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-12T21:50:39.344 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:39 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:50:39.138+0000 7fa7bbc13000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-12T21:50:39.344 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:39 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:50:39.300+0000 7fa7bbc13000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-12T21:50:39.440 INFO:teuthology.orchestra.run.smithi097.stderr:Waiting for the mgr to restart... 2024-04-12T21:50:39.441 INFO:teuthology.orchestra.run.smithi097.stderr:Waiting for mgr epoch 9... 2024-04-12T21:50:39.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:39 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/3730074547' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-12T21:50:39.661 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:39 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:50:39.387+0000 7fa7bbc13000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-12T21:50:40.771 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:40 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:50:40.444+0000 7fa7bbc13000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-12T21:50:40.772 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:40 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:50:40.576+0000 7fa7bbc13000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-12T21:50:40.772 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:40 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:50:40.770+0000 7fa7bbc13000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-12T21:50:41.161 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:40 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:50:40.858+0000 7fa7bbc13000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-12T21:50:41.161 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:40 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:50:40.996+0000 7fa7bbc13000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-12T21:50:41.615 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:41 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:50:41.311+0000 7fa7bbc13000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-12T21:50:41.615 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:41 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:50:41.530+0000 7fa7bbc13000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-12T21:50:41.615 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:41 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:50:41.613+0000 7fa7bbc13000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-12T21:50:41.910 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:41 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:50:41.782+0000 7fa7bbc13000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-12T21:50:42.992 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:42 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:50:42.699+0000 7fa7bbc13000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-12T21:50:42.992 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:42 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:50:42.991+0000 7fa7bbc13000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-12T21:50:43.411 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:43 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:50:43.085+0000 7fa7bbc13000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-12T21:50:43.411 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:43 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:50:43.233+0000 7fa7bbc13000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-12T21:50:44.325 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:44 smithi097 ceph-mon[24835]: Active manager daemon y restarted 2024-04-12T21:50:44.326 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:44 smithi097 ceph-mon[24835]: Activating manager daemon y 2024-04-12T21:50:44.326 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:44 smithi097 ceph-mon[24835]: osdmap e3: 0 total, 0 up, 0 in 2024-04-12T21:50:45.081 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: { 2024-04-12T21:50:45.081 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:45 smithi097 ceph-mon[24835]: mgrmap e10: y(active, starting, since 0.827502s) 2024-04-12T21:50:45.081 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:45 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-12T21:50:45.081 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:45 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-12T21:50:45.081 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:45 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-12T21:50:45.082 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:45 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-12T21:50:45.082 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:45 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-12T21:50:45.082 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:45 smithi097 ceph-mon[24835]: Manager daemon y is now available 2024-04-12T21:50:45.082 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:45 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:50:45.082 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:45 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:50:45.082 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:45 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T21:50:45.082 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:45 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T21:50:45.082 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-04-12T21:50:45.083 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: "initialized": true 2024-04-12T21:50:45.083 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: } 2024-04-12T21:50:45.411 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:45 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: [12/Apr/2024:21:50:45] ENGINE Bus STARTING 2024-04-12T21:50:45.411 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:45 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: [12/Apr/2024:21:50:45] ENGINE Serving on https://172.21.15.97:7150 2024-04-12T21:50:45.411 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:50:45 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: [12/Apr/2024:21:50:45] ENGINE Bus STARTED 2024-04-12T21:50:45.768 INFO:teuthology.orchestra.run.smithi097.stderr:mgr epoch 9 is available 2024-04-12T21:50:45.769 INFO:teuthology.orchestra.run.smithi097.stderr:Generating a dashboard self-signed certificate... 2024-04-12T21:50:46.357 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:46 smithi097 ceph-mon[24835]: mgrmap e11: y(active, since 1.83372s) 2024-04-12T21:50:46.357 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:46 smithi097 ceph-mon[24835]: [12/Apr/2024:21:50:45] ENGINE Bus STARTING 2024-04-12T21:50:46.357 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:46 smithi097 ceph-mon[24835]: [12/Apr/2024:21:50:45] ENGINE Serving on https://172.21.15.97:7150 2024-04-12T21:50:46.357 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:46 smithi097 ceph-mon[24835]: [12/Apr/2024:21:50:45] ENGINE Bus STARTED 2024-04-12T21:50:46.357 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:46 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:50:46.546 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: Self-signed certificate created 2024-04-12T21:50:47.268 INFO:teuthology.orchestra.run.smithi097.stderr:Creating initial admin user... 2024-04-12T21:50:47.533 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:47 smithi097 ceph-mon[24835]: mgrmap e12: y(active, since 2s) 2024-04-12T21:50:47.533 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:47 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:50:47.533 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:47 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:50:47.534 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:47 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:50:48.204 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$UFYB0GhYNA57YYE/ZxHxmuVV6L0gn.fovknKhBZ4ol253a90bi9o2", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1712958648, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-12T21:50:48.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:48 smithi097 ceph-mon[24835]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:50:48.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:48 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:50:48.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:48 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:50:48.886 INFO:teuthology.orchestra.run.smithi097.stderr:Fetching dashboard port number... 2024-04-12T21:50:49.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:49 smithi097 ceph-mon[24835]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:50:49.519 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: 8443 2024-04-12T21:50:50.301 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:50 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/633195882' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-12T21:50:50.700 INFO:teuthology.orchestra.run.smithi097.stderr:systemctl: disabled 2024-04-12T21:50:50.707 INFO:teuthology.orchestra.run.smithi097.stderr:systemctl: inactive 2024-04-12T21:50:50.708 INFO:teuthology.orchestra.run.smithi097.stderr:firewalld.service is not enabled 2024-04-12T21:50:50.708 INFO:teuthology.orchestra.run.smithi097.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-12T21:50:50.710 INFO:teuthology.orchestra.run.smithi097.stderr:Ceph Dashboard is now available at: 2024-04-12T21:50:50.710 INFO:teuthology.orchestra.run.smithi097.stderr: 2024-04-12T21:50:50.710 INFO:teuthology.orchestra.run.smithi097.stderr: URL: https://smithi097.front.sepia.ceph.com:8443/ 2024-04-12T21:50:50.710 INFO:teuthology.orchestra.run.smithi097.stderr: User: admin 2024-04-12T21:50:50.711 INFO:teuthology.orchestra.run.smithi097.stderr: Password: 4pzf4bi01n 2024-04-12T21:50:50.711 INFO:teuthology.orchestra.run.smithi097.stderr: 2024-04-12T21:50:50.711 INFO:teuthology.orchestra.run.smithi097.stderr:Enabling autotune for osd_memory_target 2024-04-12T21:50:51.259 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:51 smithi097 ceph-mon[24835]: mgrmap e13: y(active, since 6s) 2024-04-12T21:50:52.770 INFO:teuthology.orchestra.run.smithi097.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-04-12T21:50:53.447 INFO:teuthology.orchestra.run.smithi097.stderr:You can access the Ceph CLI with: 2024-04-12T21:50:53.447 INFO:teuthology.orchestra.run.smithi097.stderr: 2024-04-12T21:50:53.447 INFO:teuthology.orchestra.run.smithi097.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-12T21:50:53.447 INFO:teuthology.orchestra.run.smithi097.stderr: 2024-04-12T21:50:53.447 INFO:teuthology.orchestra.run.smithi097.stderr:Please consider enabling telemetry to help improve Ceph: 2024-04-12T21:50:53.447 INFO:teuthology.orchestra.run.smithi097.stderr: 2024-04-12T21:50:53.447 INFO:teuthology.orchestra.run.smithi097.stderr: ceph telemetry on 2024-04-12T21:50:53.447 INFO:teuthology.orchestra.run.smithi097.stderr: 2024-04-12T21:50:53.448 INFO:teuthology.orchestra.run.smithi097.stderr:For more information see: 2024-04-12T21:50:53.448 INFO:teuthology.orchestra.run.smithi097.stderr: 2024-04-12T21:50:53.448 INFO:teuthology.orchestra.run.smithi097.stderr: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-04-12T21:50:53.448 INFO:teuthology.orchestra.run.smithi097.stderr: 2024-04-12T21:50:53.448 INFO:teuthology.orchestra.run.smithi097.stderr:Bootstrap complete. 2024-04-12T21:50:53.485 INFO:tasks.cephadm:Fetching config... 2024-04-12T21:50:53.485 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-12T21:50:53.486 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-12T21:50:53.502 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-12T21:50:53.502 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-12T21:50:53.502 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-12T21:50:53.557 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-12T21:50:53.557 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-12T21:50:53.558 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-12T21:50:53.625 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-12T21:50:53.625 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-12T21:50:53.626 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-12T21:50:53.680 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-12T21:50:53.681 DEBUG:teuthology.orchestra.run.smithi097:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCuO/R8dfmju8cc46G8LpiLuesKcxRF24sV0lpnRQ7vYOUop74Btl4FF1Khi8iWfUJ3GG2YlxkAOc76SNs3f+ko6u5mMcNzAkerP6vPD7GFO4PNBVrOmc6CeWmSycHZAoLkppLcMIFz6fpXgMu2ekqrGqozWjUgVY0C7Zc4kEUojm9z3yxGaCQYb8Mcm0znqRK1pjfHRjjLrnEG1t/lWA0odg0RwxQm0imuTCitlekFp+d8kNu6BC0XhGrro65J9zmrVWgCfpkE0aUqwRauSvaE04actKG8Hml3/Buu4q5p2XU45rsLS3GTcZ4dZVPm+nj1IvOCaqv0tBA6XUEWuDq20m5GKPdd7tla2CUh2iW34cfwBwjMRso+HhpQDXLQLNJSvCHIG6sOTpPimANJjC22xsnd1kCXkOU/cKxBiFJQS697kT09BfEuu9pWSRAk7l5GZjay4Q+BO0AvG/aeuowjbtkG3FFlHrBESTcvMZkut3W894H51ckMi6mv6nHczGE= ceph-80bf7680-f916-11ee-bc8d-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-12T21:50:53.760 INFO:teuthology.orchestra.run.smithi097.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCuO/R8dfmju8cc46G8LpiLuesKcxRF24sV0lpnRQ7vYOUop74Btl4FF1Khi8iWfUJ3GG2YlxkAOc76SNs3f+ko6u5mMcNzAkerP6vPD7GFO4PNBVrOmc6CeWmSycHZAoLkppLcMIFz6fpXgMu2ekqrGqozWjUgVY0C7Zc4kEUojm9z3yxGaCQYb8Mcm0znqRK1pjfHRjjLrnEG1t/lWA0odg0RwxQm0imuTCitlekFp+d8kNu6BC0XhGrro65J9zmrVWgCfpkE0aUqwRauSvaE04actKG8Hml3/Buu4q5p2XU45rsLS3GTcZ4dZVPm+nj1IvOCaqv0tBA6XUEWuDq20m5GKPdd7tla2CUh2iW34cfwBwjMRso+HhpQDXLQLNJSvCHIG6sOTpPimANJjC22xsnd1kCXkOU/cKxBiFJQS697kT09BfEuu9pWSRAk7l5GZjay4Q+BO0AvG/aeuowjbtkG3FFlHrBESTcvMZkut3W894H51ckMi6mv6nHczGE= ceph-80bf7680-f916-11ee-bc8d-c7b262605968 2024-04-12T21:50:53.774 DEBUG:teuthology.orchestra.run.smithi114:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCuO/R8dfmju8cc46G8LpiLuesKcxRF24sV0lpnRQ7vYOUop74Btl4FF1Khi8iWfUJ3GG2YlxkAOc76SNs3f+ko6u5mMcNzAkerP6vPD7GFO4PNBVrOmc6CeWmSycHZAoLkppLcMIFz6fpXgMu2ekqrGqozWjUgVY0C7Zc4kEUojm9z3yxGaCQYb8Mcm0znqRK1pjfHRjjLrnEG1t/lWA0odg0RwxQm0imuTCitlekFp+d8kNu6BC0XhGrro65J9zmrVWgCfpkE0aUqwRauSvaE04actKG8Hml3/Buu4q5p2XU45rsLS3GTcZ4dZVPm+nj1IvOCaqv0tBA6XUEWuDq20m5GKPdd7tla2CUh2iW34cfwBwjMRso+HhpQDXLQLNJSvCHIG6sOTpPimANJjC22xsnd1kCXkOU/cKxBiFJQS697kT09BfEuu9pWSRAk7l5GZjay4Q+BO0AvG/aeuowjbtkG3FFlHrBESTcvMZkut3W894H51ckMi6mv6nHczGE= ceph-80bf7680-f916-11ee-bc8d-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-12T21:50:53.823 INFO:teuthology.orchestra.run.smithi114.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCuO/R8dfmju8cc46G8LpiLuesKcxRF24sV0lpnRQ7vYOUop74Btl4FF1Khi8iWfUJ3GG2YlxkAOc76SNs3f+ko6u5mMcNzAkerP6vPD7GFO4PNBVrOmc6CeWmSycHZAoLkppLcMIFz6fpXgMu2ekqrGqozWjUgVY0C7Zc4kEUojm9z3yxGaCQYb8Mcm0znqRK1pjfHRjjLrnEG1t/lWA0odg0RwxQm0imuTCitlekFp+d8kNu6BC0XhGrro65J9zmrVWgCfpkE0aUqwRauSvaE04actKG8Hml3/Buu4q5p2XU45rsLS3GTcZ4dZVPm+nj1IvOCaqv0tBA6XUEWuDq20m5GKPdd7tla2CUh2iW34cfwBwjMRso+HhpQDXLQLNJSvCHIG6sOTpPimANJjC22xsnd1kCXkOU/cKxBiFJQS697kT09BfEuu9pWSRAk7l5GZjay4Q+BO0AvG/aeuowjbtkG3FFlHrBESTcvMZkut3W894H51ckMi6mv6nHczGE= ceph-80bf7680-f916-11ee-bc8d-c7b262605968 2024-04-12T21:50:53.837 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-12T21:50:54.145 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:53 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/2096426961' entity='client.admin' 2024-04-12T21:50:57.646 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-12T21:50:57.646 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-12T21:50:58.386 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:58 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/2170344315' entity='client.admin' 2024-04-12T21:50:59.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:59 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:50:59.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:59 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi097", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:50:59.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:59 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:50:59.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:59 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:50:59.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:50:59 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:00.705 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi114 2024-04-12T21:51:00.706 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-04-12T21:51:00.706 DEBUG:teuthology.orchestra.run.smithi114:> dd of=/etc/ceph/ceph.conf 2024-04-12T21:51:00.724 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-04-12T21:51:00.724 DEBUG:teuthology.orchestra.run.smithi114:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-12T21:51:00.782 INFO:tasks.cephadm:Adding host smithi114 to orchestrator... 2024-04-12T21:51:00.782 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- ceph orch host add smithi114 2024-04-12T21:51:01.319 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:00 smithi097 ceph-mon[24835]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:51:01.320 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:00 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:01.320 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:00 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:51:01.320 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:00 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:51:01.320 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:00 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:51:01.320 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:00 smithi097 ceph-mon[24835]: Updating smithi097:/etc/ceph/ceph.conf 2024-04-12T21:51:01.320 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:00 smithi097 ceph-mon[24835]: Updating smithi097:/etc/ceph/ceph.client.admin.keyring 2024-04-12T21:51:01.320 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:00 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:01.320 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:00 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:03.641 INFO:teuthology.orchestra.run.smithi097.stdout:Added host 'smithi114' with addr '172.21.15.114' 2024-04-12T21:51:04.211 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:04 smithi097 ceph-mon[24835]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi114", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:51:04.211 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:04 smithi097 ceph-mon[24835]: Deploying cephadm binary to smithi114 2024-04-12T21:51:04.211 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:04 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:04.211 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:04 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:51:04.212 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:04 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:51:04.212 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:04 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:51:04.212 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:04 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:04.213 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- ceph orch host ls --format=json 2024-04-12T21:51:05.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:05 smithi097 ceph-mon[24835]: Added host smithi114 2024-04-12T21:51:05.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:05 smithi097 ceph-mon[24835]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:05.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:05 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:06.090 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-12T21:51:06.090 INFO:teuthology.orchestra.run.smithi097.stdout:[{"addr": "172.21.15.97", "hostname": "smithi097", "labels": [], "status": ""}, {"addr": "172.21.15.114", "hostname": "smithi114", "labels": [], "status": ""}] 2024-04-12T21:51:06.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:06 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:06.701 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-12T21:51:06.701 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- ceph osd crush tunables default 2024-04-12T21:51:07.278 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:07 smithi097 ceph-mon[24835]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:07.279 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:07 smithi097 ceph-mon[24835]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-12T21:51:09.178 INFO:teuthology.orchestra.run.smithi097.stderr:adjusted tunables profile to default 2024-04-12T21:51:09.505 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:09 smithi097 ceph-mon[24835]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:09.505 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:09 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:09.506 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:09 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:09.506 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:09 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi114", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:51:09.506 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:09 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:09.506 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:09 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/139514070' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-12T21:51:09.746 INFO:tasks.cephadm:Adding mon.a on smithi097 2024-04-12T21:51:09.746 INFO:tasks.cephadm:Adding mon.c on smithi097 2024-04-12T21:51:09.746 INFO:tasks.cephadm:Adding mon.b on smithi114 2024-04-12T21:51:09.746 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- ceph orch apply mon '3;smithi097:172.21.15.97=a;smithi097:[v2:172.21.15.97:3301,v1:172.21.15.97:6790]=c;smithi114:172.21.15.114=b' 2024-04-12T21:51:10.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:10 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/139514070' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-12T21:51:10.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:10 smithi097 ceph-mon[24835]: osdmap e4: 0 total, 0 up, 0 in 2024-04-12T21:51:11.432 INFO:teuthology.orchestra.run.smithi114.stdout:Scheduled mon update... 2024-04-12T21:51:11.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:11 smithi097 ceph-mon[24835]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:12.165 DEBUG:teuthology.orchestra.run.smithi097:mon.c> sudo journalctl -f -n 0 -u ceph-80bf7680-f916-11ee-bc8d-c7b262605968@mon.c.service 2024-04-12T21:51:12.170 DEBUG:teuthology.orchestra.run.smithi114:mon.b> sudo journalctl -f -n 0 -u ceph-80bf7680-f916-11ee-bc8d-c7b262605968@mon.b.service 2024-04-12T21:51:12.173 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-12T21:51:12.173 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- ceph mon dump -f json 2024-04-12T21:51:12.659 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:12 smithi097 ceph-mon[24835]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi097:172.21.15.97=a;smithi097:[v2:172.21.15.97:3301,v1:172.21.15.97:6790]=c;smithi114:172.21.15.114=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:51:12.659 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:12 smithi097 ceph-mon[24835]: Saving service mon spec with placement smithi097:172.21.15.97=a;smithi097:[v2:172.21.15.97:3301,v1:172.21.15.97:6790]=c;smithi114:172.21.15.114=b;count:3 2024-04-12T21:51:12.659 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:12 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:12.659 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:12 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:51:12.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:12 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:51:12.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:12 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:51:12.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:12 smithi097 ceph-mon[24835]: Updating smithi114:/etc/ceph/ceph.conf 2024-04-12T21:51:12.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:12 smithi097 ceph-mon[24835]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-04-12T21:51:12.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:12 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:12.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:12 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:12.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:12 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T21:51:12.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:12 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:51:12.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:12 smithi097 ceph-mon[24835]: Deploying daemon mon.c on smithi097 2024-04-12T21:51:12.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:12 smithi097 ceph-mon[24835]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:14.064 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-12T21:51:14.064 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"80bf7680-f916-11ee-bc8d-c7b262605968","modified":"2024-04-12T21:49:45.534140Z","created":"2024-04-12T21:49:45.534140Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:3300","nonce":0},{"type":"v1","addr":"172.21.15.97:6789","nonce":0}]},"addr":"172.21.15.97:6789/0","public_addr":"172.21.15.97:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-12T21:51:14.066 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-04-12T21:51:14.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:14 smithi097 ceph-mon[24835]: from='client.? 172.21.15.114:0/370944443' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-12T21:51:15.122 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 podman[29601]: 2024-04-12 21:51:15.068591564 +0000 UTC m=+0.126831284 container create 679b33a29a0cc1a76fb83b8403c75c795f938108068da55843d41c2882859362 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mon-c, io.openshift.tags=base centos centos-stream, ceph=True, name=centos-stream, maintainer=Guillaume Abrioux , RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, version=8, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, vcs-type=git, com.redhat.component=centos-stream-container, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., architecture=x86_64) 2024-04-12T21:51:15.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[24835]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:15.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 podman[29601]: 2024-04-12 21:51:15.246058486 +0000 UTC m=+0.304298208 container init 679b33a29a0cc1a76fb83b8403c75c795f938108068da55843d41c2882859362 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mon-c, GIT_CLEAN=True, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, io.buildah.version=1.19.8, RELEASE=HEAD, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64) 2024-04-12T21:51:15.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 podman[29601]: 2024-04-12 21:51:15.248943141 +0000 UTC m=+0.307182863 container start 679b33a29a0cc1a76fb83b8403c75c795f938108068da55843d41c2882859362 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mon-c, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, ceph=True, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, io.buildah.version=1.19.8, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-04-12T21:51:15.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: set uid:gid to 167:167 (ceph:ceph) 2024-04-12T21:51:15.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-04-12T21:51:15.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: pidfile_write: ignore empty --pid-file 2024-04-12T21:51:15.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: load: jerasure load: lrc 2024-04-12T21:51:15.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: RocksDB version: 6.15.5 2024-04-12T21:51:15.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-04-12T21:51:15.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Compile date Apr 18 2022 2024-04-12T21:51:15.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: DB SUMMARY 2024-04-12T21:51:15.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: DB Session ID: K1LH0KQSVAM95E820FLQ 2024-04-12T21:51:15.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: CURRENT file: CURRENT 2024-04-12T21:51:15.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: IDENTITY file: IDENTITY 2024-04-12T21:51:15.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-04-12T21:51:15.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-04-12T21:51:15.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-04-12T21:51:15.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.error_if_exists: 0 2024-04-12T21:51:15.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.create_if_missing: 0 2024-04-12T21:51:15.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.paranoid_checks: 1 2024-04-12T21:51:15.413 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-12T21:51:15.413 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.env: 0x55a751ac6860 2024-04-12T21:51:15.413 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.fs: Posix File System 2024-04-12T21:51:15.413 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.info_log: 0x55a75302ddc0 2024-04-12T21:51:15.413 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.max_file_opening_threads: 16 2024-04-12T21:51:15.413 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.statistics: (nil) 2024-04-12T21:51:15.413 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.use_fsync: 0 2024-04-12T21:51:15.413 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.max_log_file_size: 0 2024-04-12T21:51:15.413 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-12T21:51:15.413 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-12T21:51:15.413 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.keep_log_file_num: 1000 2024-04-12T21:51:15.413 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.recycle_log_file_num: 0 2024-04-12T21:51:15.413 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.allow_fallocate: 1 2024-04-12T21:51:15.413 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.allow_mmap_reads: 0 2024-04-12T21:51:15.413 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.allow_mmap_writes: 0 2024-04-12T21:51:15.414 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.use_direct_reads: 0 2024-04-12T21:51:15.414 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-12T21:51:15.414 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.create_missing_column_families: 0 2024-04-12T21:51:15.414 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.db_log_dir: 2024-04-12T21:51:15.414 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-c/store.db 2024-04-12T21:51:15.414 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-12T21:51:15.414 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-12T21:51:15.414 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-12T21:51:15.414 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-12T21:51:15.414 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-12T21:51:15.414 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-12T21:51:15.416 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.advise_random_on_open: 1 2024-04-12T21:51:15.416 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.db_write_buffer_size: 0 2024-04-12T21:51:15.416 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.write_buffer_manager: 0x55a75311e2d0 2024-04-12T21:51:15.416 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-12T21:51:15.416 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-04-12T21:51:15.416 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-12T21:51:15.416 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-12T21:51:15.416 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.rate_limiter: (nil) 2024-04-12T21:51:15.416 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-12T21:51:15.416 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.wal_recovery_mode: 2 2024-04-12T21:51:15.416 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.enable_thread_tracking: 0 2024-04-12T21:51:15.417 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.enable_pipelined_write: 0 2024-04-12T21:51:15.417 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.unordered_write: 0 2024-04-12T21:51:15.417 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-12T21:51:15.417 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-12T21:51:15.417 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-12T21:51:15.417 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-12T21:51:15.417 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.row_cache: None 2024-04-12T21:51:15.417 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.wal_filter: None 2024-04-12T21:51:15.417 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-12T21:51:15.417 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.allow_ingest_behind: 0 2024-04-12T21:51:15.417 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.preserve_deletes: 0 2024-04-12T21:51:15.417 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.two_write_queues: 0 2024-04-12T21:51:15.417 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.manual_wal_flush: 0 2024-04-12T21:51:15.417 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.atomic_flush: 0 2024-04-12T21:51:15.418 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-12T21:51:15.418 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-12T21:51:15.418 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-12T21:51:15.418 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.log_readahead_size: 0 2024-04-12T21:51:15.418 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-12T21:51:15.418 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.best_efforts_recovery: 0 2024-04-12T21:51:15.418 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-12T21:51:15.418 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-12T21:51:15.418 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.allow_data_in_errors: 0 2024-04-12T21:51:15.418 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.db_host_id: __hostname__ 2024-04-12T21:51:15.418 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.max_background_jobs: 2 2024-04-12T21:51:15.418 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.max_background_compactions: -1 2024-04-12T21:51:15.418 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.max_subcompactions: 1 2024-04-12T21:51:15.418 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-12T21:51:15.418 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-12T21:51:15.419 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-12T21:51:15.419 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.max_total_wal_size: 0 2024-04-12T21:51:15.419 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-12T21:51:15.419 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-12T21:51:15.419 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-12T21:51:15.419 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-12T21:51:15.419 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.max_open_files: -1 2024-04-12T21:51:15.419 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.bytes_per_sync: 0 2024-04-12T21:51:15.419 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-12T21:51:15.419 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-12T21:51:15.419 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.compaction_readahead_size: 0 2024-04-12T21:51:15.419 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.max_background_flushes: -1 2024-04-12T21:51:15.419 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Compression algorithms supported: 2024-04-12T21:51:15.419 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-12T21:51:15.420 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: kZSTD supported: 0 2024-04-12T21:51:15.420 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: kXpressCompression supported: 0 2024-04-12T21:51:15.420 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: kLZ4HCCompression supported: 1 2024-04-12T21:51:15.420 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: kLZ4Compression supported: 1 2024-04-12T21:51:15.420 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: kBZip2Compression supported: 0 2024-04-12T21:51:15.420 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: kZlibCompression supported: 1 2024-04-12T21:51:15.420 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: kSnappyCompression supported: 1 2024-04-12T21:51:15.420 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-12T21:51:15.420 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 2024-04-12T21:51:15.420 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-04-12T21:51:15.420 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-12T21:51:15.420 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.merge_operator: 2024-04-12T21:51:15.420 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.compaction_filter: None 2024-04-12T21:51:15.420 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.compaction_filter_factory: None 2024-04-12T21:51:15.420 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.sst_partitioner_factory: None 2024-04-12T21:51:15.421 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-12T21:51:15.421 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-12T21:51:15.421 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55a752ff9d00) 2024-04-12T21:51:15.421 INFO:journalctl@ceph.mon.c.smithi097.stdout: cache_index_and_filter_blocks: 1 2024-04-12T21:51:15.421 INFO:journalctl@ceph.mon.c.smithi097.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-12T21:51:15.421 INFO:journalctl@ceph.mon.c.smithi097.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-12T21:51:15.421 INFO:journalctl@ceph.mon.c.smithi097.stdout: pin_top_level_index_and_filter: 1 2024-04-12T21:51:15.421 INFO:journalctl@ceph.mon.c.smithi097.stdout: index_type: 0 2024-04-12T21:51:15.421 INFO:journalctl@ceph.mon.c.smithi097.stdout: data_block_index_type: 0 2024-04-12T21:51:15.421 INFO:journalctl@ceph.mon.c.smithi097.stdout: index_shortening: 1 2024-04-12T21:51:15.421 INFO:journalctl@ceph.mon.c.smithi097.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-12T21:51:15.421 INFO:journalctl@ceph.mon.c.smithi097.stdout: hash_index_allow_collision: 1 2024-04-12T21:51:15.421 INFO:journalctl@ceph.mon.c.smithi097.stdout: checksum: 1 2024-04-12T21:51:15.421 INFO:journalctl@ceph.mon.c.smithi097.stdout: no_block_cache: 0 2024-04-12T21:51:15.421 INFO:journalctl@ceph.mon.c.smithi097.stdout: block_cache: 0x55a753064170 2024-04-12T21:51:15.422 INFO:journalctl@ceph.mon.c.smithi097.stdout: block_cache_name: BinnedLRUCache 2024-04-12T21:51:15.422 INFO:journalctl@ceph.mon.c.smithi097.stdout: block_cache_options: 2024-04-12T21:51:15.422 INFO:journalctl@ceph.mon.c.smithi097.stdout: capacity : 536870912 2024-04-12T21:51:15.422 INFO:journalctl@ceph.mon.c.smithi097.stdout: num_shard_bits : 4 2024-04-12T21:51:15.422 INFO:journalctl@ceph.mon.c.smithi097.stdout: strict_capacity_limit : 0 2024-04-12T21:51:15.422 INFO:journalctl@ceph.mon.c.smithi097.stdout: high_pri_pool_ratio: 0.000 2024-04-12T21:51:15.422 INFO:journalctl@ceph.mon.c.smithi097.stdout: block_cache_compressed: (nil) 2024-04-12T21:51:15.422 INFO:journalctl@ceph.mon.c.smithi097.stdout: persistent_cache: (nil) 2024-04-12T21:51:15.422 INFO:journalctl@ceph.mon.c.smithi097.stdout: block_size: 4096 2024-04-12T21:51:15.422 INFO:journalctl@ceph.mon.c.smithi097.stdout: block_size_deviation: 10 2024-04-12T21:51:15.422 INFO:journalctl@ceph.mon.c.smithi097.stdout: block_restart_interval: 16 2024-04-12T21:51:15.422 INFO:journalctl@ceph.mon.c.smithi097.stdout: index_block_restart_interval: 1 2024-04-12T21:51:15.422 INFO:journalctl@ceph.mon.c.smithi097.stdout: metadata_block_size: 4096 2024-04-12T21:51:15.422 INFO:journalctl@ceph.mon.c.smithi097.stdout: partition_filters: 0 2024-04-12T21:51:15.423 INFO:journalctl@ceph.mon.c.smithi097.stdout: use_delta_encoding: 1 2024-04-12T21:51:15.423 INFO:journalctl@ceph.mon.c.smithi097.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-04-12T21:51:15.423 INFO:journalctl@ceph.mon.c.smithi097.stdout: whole_key_filtering: 1 2024-04-12T21:51:15.423 INFO:journalctl@ceph.mon.c.smithi097.stdout: verify_compression: 0 2024-04-12T21:51:15.423 INFO:journalctl@ceph.mon.c.smithi097.stdout: read_amp_bytes_per_bit: 0 2024-04-12T21:51:15.423 INFO:journalctl@ceph.mon.c.smithi097.stdout: format_version: 4 2024-04-12T21:51:15.423 INFO:journalctl@ceph.mon.c.smithi097.stdout: enable_index_compression: 1 2024-04-12T21:51:15.423 INFO:journalctl@ceph.mon.c.smithi097.stdout: block_align: 0 2024-04-12T21:51:15.423 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.write_buffer_size: 33554432 2024-04-12T21:51:15.423 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.max_write_buffer_number: 2 2024-04-12T21:51:15.423 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.compression: NoCompression 2024-04-12T21:51:15.423 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.bottommost_compression: Disabled 2024-04-12T21:51:15.423 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.prefix_extractor: nullptr 2024-04-12T21:51:15.423 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-12T21:51:15.423 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.num_levels: 7 2024-04-12T21:51:15.423 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-12T21:51:15.424 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-12T21:51:15.424 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-12T21:51:15.424 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-12T21:51:15.424 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-12T21:51:15.424 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-12T21:51:15.424 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-12T21:51:15.424 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-12T21:51:15.424 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-12T21:51:15.424 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-12T21:51:15.424 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-12T21:51:15.424 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.compression_opts.level: 32767 2024-04-12T21:51:15.424 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.compression_opts.strategy: 0 2024-04-12T21:51:15.424 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-12T21:51:15.424 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-12T21:51:15.424 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-12T21:51:15.425 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.compression_opts.enabled: false 2024-04-12T21:51:15.425 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-12T21:51:15.425 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-12T21:51:15.425 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-12T21:51:15.425 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.target_file_size_base: 67108864 2024-04-12T21:51:15.425 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-12T21:51:15.425 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-12T21:51:15.425 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-12T21:51:15.425 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-12T21:51:15.425 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-12T21:51:15.425 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-12T21:51:15.425 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-12T21:51:15.425 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-12T21:51:15.425 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-12T21:51:15.426 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-12T21:51:15.426 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-12T21:51:15.426 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-12T21:51:15.426 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-12T21:51:15.426 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.arena_block_size: 4194304 2024-04-12T21:51:15.426 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-12T21:51:15.426 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-12T21:51:15.426 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-04-12T21:51:15.426 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.disable_auto_compactions: 0 2024-04-12T21:51:15.426 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-12T21:51:15.426 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-12T21:51:15.426 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-12T21:51:15.426 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-12T21:51:15.426 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-12T21:51:15.427 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-12T21:51:15.427 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-12T21:51:15.427 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-12T21:51:15.427 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-12T21:51:15.427 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-12T21:51:15.427 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.table_properties_collectors: 2024-04-12T21:51:15.427 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.inplace_update_support: 0 2024-04-12T21:51:15.427 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-12T21:51:15.427 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-12T21:51:15.427 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-12T21:51:15.427 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-12T21:51:15.427 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.bloom_locality: 0 2024-04-12T21:51:15.427 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.max_successive_merges: 0 2024-04-12T21:51:15.427 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-12T21:51:15.428 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.paranoid_file_checks: 0 2024-04-12T21:51:15.428 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.force_consistency_checks: 1 2024-04-12T21:51:15.428 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.report_bg_io_stats: 0 2024-04-12T21:51:15.428 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.ttl: 2592000 2024-04-12T21:51:15.428 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-12T21:51:15.428 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.enable_blob_files: false 2024-04-12T21:51:15.428 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.min_blob_size: 0 2024-04-12T21:51:15.428 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.blob_file_size: 268435456 2024-04-12T21:51:15.428 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-12T21:51:15.428 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-12T21:51:15.428 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-12T21:51:15.428 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-12T21:51:15.428 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 0 2024-04-12T21:51:15.428 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: [db/version_set.cc:4083] Creating manifest 7 2024-04-12T21:51:15.428 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712958675274534, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-12T21:51:15.429 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #4 mode 2 2024-04-12T21:51:15.429 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-04-12T21:51:15.429 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712958675274830, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1545, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1712958675, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "843dc4ac-3e8d-43fe-833b-e7956a78dc14", "db_session_id": "K1LH0KQSVAM95E820FLQ"}} 2024-04-12T21:51:15.429 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: [db/version_set.cc:4083] Creating manifest 9 2024-04-12T21:51:15.429 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712958675275163, "job": 1, "event": "recovery_finished"} 2024-04-12T21:51:15.429 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-12T21:51:15.429 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x55a75304aa80 2024-04-12T21:51:15.429 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: DB pointer 0x55a75305a000 2024-04-12T21:51:15.429 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-04-12T21:51:15.429 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-04-12T21:51:15.429 INFO:journalctl@ceph.mon.c.smithi097.stdout: ** DB Stats ** 2024-04-12T21:51:15.429 INFO:journalctl@ceph.mon.c.smithi097.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-12T21:51:15.429 INFO:journalctl@ceph.mon.c.smithi097.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-12T21:51:15.429 INFO:journalctl@ceph.mon.c.smithi097.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-12T21:51:15.429 INFO:journalctl@ceph.mon.c.smithi097.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-12T21:51:15.430 INFO:journalctl@ceph.mon.c.smithi097.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-12T21:51:15.430 INFO:journalctl@ceph.mon.c.smithi097.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-04-12T21:51:15.430 INFO:journalctl@ceph.mon.c.smithi097.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-12T21:51:15.430 INFO:journalctl@ceph.mon.c.smithi097.stdout: 2024-04-12T21:51:15.430 INFO:journalctl@ceph.mon.c.smithi097.stdout: ** Compaction Stats [default] ** 2024-04-12T21:51:15.430 INFO:journalctl@ceph.mon.c.smithi097.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-12T21:51:15.430 INFO:journalctl@ceph.mon.c.smithi097.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-12T21:51:15.430 INFO:journalctl@ceph.mon.c.smithi097.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-04-12T21:51:15.430 INFO:journalctl@ceph.mon.c.smithi097.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-04-12T21:51:15.430 INFO:journalctl@ceph.mon.c.smithi097.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-04-12T21:51:15.430 INFO:journalctl@ceph.mon.c.smithi097.stdout: 2024-04-12T21:51:15.430 INFO:journalctl@ceph.mon.c.smithi097.stdout: ** Compaction Stats [default] ** 2024-04-12T21:51:15.430 INFO:journalctl@ceph.mon.c.smithi097.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-12T21:51:15.430 INFO:journalctl@ceph.mon.c.smithi097.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-12T21:51:15.430 INFO:journalctl@ceph.mon.c.smithi097.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-04-12T21:51:15.431 INFO:journalctl@ceph.mon.c.smithi097.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-12T21:51:15.431 INFO:journalctl@ceph.mon.c.smithi097.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-12T21:51:15.431 INFO:journalctl@ceph.mon.c.smithi097.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-12T21:51:15.431 INFO:journalctl@ceph.mon.c.smithi097.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-12T21:51:15.431 INFO:journalctl@ceph.mon.c.smithi097.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-12T21:51:15.431 INFO:journalctl@ceph.mon.c.smithi097.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-12T21:51:15.431 INFO:journalctl@ceph.mon.c.smithi097.stdout: Cumulative compaction: 0.00 GB write, 0.60 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-12T21:51:15.431 INFO:journalctl@ceph.mon.c.smithi097.stdout: Interval compaction: 0.00 GB write, 0.60 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-12T21:51:15.431 INFO:journalctl@ceph.mon.c.smithi097.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-12T21:51:15.431 INFO:journalctl@ceph.mon.c.smithi097.stdout: 2024-04-12T21:51:15.431 INFO:journalctl@ceph.mon.c.smithi097.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-12T21:51:15.431 INFO:journalctl@ceph.mon.c.smithi097.stdout: 2024-04-12T21:51:15.431 INFO:journalctl@ceph.mon.c.smithi097.stdout: ** Compaction Stats [default] ** 2024-04-12T21:51:15.431 INFO:journalctl@ceph.mon.c.smithi097.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-12T21:51:15.432 INFO:journalctl@ceph.mon.c.smithi097.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-12T21:51:15.432 INFO:journalctl@ceph.mon.c.smithi097.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-04-12T21:51:15.432 INFO:journalctl@ceph.mon.c.smithi097.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-04-12T21:51:15.432 INFO:journalctl@ceph.mon.c.smithi097.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-04-12T21:51:15.432 INFO:journalctl@ceph.mon.c.smithi097.stdout: 2024-04-12T21:51:15.432 INFO:journalctl@ceph.mon.c.smithi097.stdout: ** Compaction Stats [default] ** 2024-04-12T21:51:15.432 INFO:journalctl@ceph.mon.c.smithi097.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-12T21:51:15.432 INFO:journalctl@ceph.mon.c.smithi097.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-12T21:51:15.432 INFO:journalctl@ceph.mon.c.smithi097.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-04-12T21:51:15.432 INFO:journalctl@ceph.mon.c.smithi097.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-12T21:51:15.432 INFO:journalctl@ceph.mon.c.smithi097.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-12T21:51:15.432 INFO:journalctl@ceph.mon.c.smithi097.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-12T21:51:15.432 INFO:journalctl@ceph.mon.c.smithi097.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-12T21:51:15.432 INFO:journalctl@ceph.mon.c.smithi097.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-12T21:51:15.432 INFO:journalctl@ceph.mon.c.smithi097.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-12T21:51:15.433 INFO:journalctl@ceph.mon.c.smithi097.stdout: Cumulative compaction: 0.00 GB write, 0.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-12T21:51:15.433 INFO:journalctl@ceph.mon.c.smithi097.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-12T21:51:15.433 INFO:journalctl@ceph.mon.c.smithi097.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-12T21:51:15.433 INFO:journalctl@ceph.mon.c.smithi097.stdout: 2024-04-12T21:51:15.433 INFO:journalctl@ceph.mon.c.smithi097.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-12T21:51:15.433 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-04-12T21:51:15.433 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: using public_addrv [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2024-04-12T21:51:15.433 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: starting mon.c rank -1 at public addrs [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] at bind addrs [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 80bf7680-f916-11ee-bc8d-c7b262605968 2024-04-12T21:51:15.433 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: mon.c@-1(???) e0 preinit fsid 80bf7680-f916-11ee-bc8d-c7b262605968 2024-04-12T21:51:15.433 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: mon.c@-1(synchronizing).mds e1 new map 2024-04-12T21:51:15.433 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: mon.c@-1(synchronizing).mds e1 print_map 2024-04-12T21:51:15.433 INFO:journalctl@ceph.mon.c.smithi097.stdout: e1 2024-04-12T21:51:15.433 INFO:journalctl@ceph.mon.c.smithi097.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T21:51:15.433 INFO:journalctl@ceph.mon.c.smithi097.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T21:51:15.433 INFO:journalctl@ceph.mon.c.smithi097.stdout: legacy client fscid: -1 2024-04-12T21:51:15.434 INFO:journalctl@ceph.mon.c.smithi097.stdout: 2024-04-12T21:51:15.434 INFO:journalctl@ceph.mon.c.smithi097.stdout: No filesystems configured 2024-04-12T21:51:15.434 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-12T21:51:15.434 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-12T21:51:15.434 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-12T21:51:15.434 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-12T21:51:15.434 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-12T21:51:15.434 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-12T21:51:15.434 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-12T21:51:15.434 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-12T21:51:15.434 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-12T21:51:15.434 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-12T21:51:15.437 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:15.437 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:15.437 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:15.437 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi114", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:51:15.438 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:15.438 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/139514070' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-12T21:51:15.438 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/139514070' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-12T21:51:15.438 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: osdmap e4: 0 total, 0 up, 0 in 2024-04-12T21:51:15.438 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:15.438 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi097:172.21.15.97=a;smithi097:[v2:172.21.15.97:3301,v1:172.21.15.97:6790]=c;smithi114:172.21.15.114=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:51:15.438 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: Saving service mon spec with placement smithi097:172.21.15.97=a;smithi097:[v2:172.21.15.97:3301,v1:172.21.15.97:6790]=c;smithi114:172.21.15.114=b;count:3 2024-04-12T21:51:15.438 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:15.438 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:51:15.438 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:51:15.438 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:51:15.438 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: Updating smithi114:/etc/ceph/ceph.conf 2024-04-12T21:51:15.438 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-04-12T21:51:15.439 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:15.439 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:15.439 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T21:51:15.439 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:51:15.439 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: Deploying daemon mon.c on smithi097 2024-04-12T21:51:15.439 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:15.439 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: from='client.? 172.21.15.114:0/370944443' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-12T21:51:15.439 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:15.439 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-12T21:51:15.439 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 ceph-mon[29620]: mon.c@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-04-12T21:51:15.439 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 bash[29601]: 679b33a29a0cc1a76fb83b8403c75c795f938108068da55843d41c2882859362 2024-04-12T21:51:15.439 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:15 smithi097 systemd[1]: Started Ceph mon.c for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T21:51:15.611 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-12T21:51:15.612 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- ceph mon dump -f json 2024-04-12T21:51:20.580 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-12T21:51:20.580 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":2,"fsid":"80bf7680-f916-11ee-bc8d-c7b262605968","modified":"2024-04-12T21:51:15.295884Z","created":"2024-04-12T21:49:45.534140Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:3300","nonce":0},{"type":"v1","addr":"172.21.15.97:6789","nonce":0}]},"addr":"172.21.15.97:6789/0","public_addr":"172.21.15.97:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:3301","nonce":0},{"type":"v1","addr":"172.21.15.97:6790","nonce":0}]},"addr":"172.21.15.97:6790/0","public_addr":"172.21.15.97:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-12T21:51:20.581 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 2 2024-04-12T21:51:20.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:20 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-12T21:51:20.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:20 smithi097 ceph-mon[29620]: mon.a calling monitor election 2024-04-12T21:51:20.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:20 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T21:51:20.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:20 smithi097 ceph-mon[29620]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:20.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:20 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T21:51:20.663 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:20 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T21:51:20.663 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:20 smithi097 ceph-mon[29620]: mon.c calling monitor election 2024-04-12T21:51:20.663 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:20 smithi097 ceph-mon[29620]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:20.663 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:20 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T21:51:20.663 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:20 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T21:51:20.664 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:20 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T21:51:20.664 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:20 smithi097 ceph-mon[29620]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-12T21:51:20.664 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:20 smithi097 ceph-mon[29620]: monmap e2: 2 mons at {a=[v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0],c=[v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0]} 2024-04-12T21:51:20.664 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:20 smithi097 ceph-mon[29620]: fsmap 2024-04-12T21:51:20.664 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:20 smithi097 ceph-mon[29620]: osdmap e4: 0 total, 0 up, 0 in 2024-04-12T21:51:20.664 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:20 smithi097 ceph-mon[29620]: mgrmap e13: y(active, since 37s) 2024-04-12T21:51:20.665 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:20 smithi097 ceph-mon[29620]: overall HEALTH_OK 2024-04-12T21:51:20.665 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:20 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:20.665 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:20 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T21:51:20.665 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:20 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:51:20.665 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:20 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-12T21:51:20.666 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:20 smithi097 ceph-mon[24835]: mon.a calling monitor election 2024-04-12T21:51:20.666 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:20 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T21:51:20.666 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:20 smithi097 ceph-mon[24835]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:20.666 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:20 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T21:51:20.666 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:20 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T21:51:20.667 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:20 smithi097 ceph-mon[24835]: mon.c calling monitor election 2024-04-12T21:51:20.667 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:20 smithi097 ceph-mon[24835]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:20.667 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:20 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T21:51:20.667 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:20 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T21:51:20.667 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:20 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T21:51:20.667 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:20 smithi097 ceph-mon[24835]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-12T21:51:20.668 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:20 smithi097 ceph-mon[24835]: monmap e2: 2 mons at {a=[v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0],c=[v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0]} 2024-04-12T21:51:20.668 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:20 smithi097 ceph-mon[24835]: fsmap 2024-04-12T21:51:20.668 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:20 smithi097 ceph-mon[24835]: osdmap e4: 0 total, 0 up, 0 in 2024-04-12T21:51:20.668 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:20 smithi097 ceph-mon[24835]: mgrmap e13: y(active, since 37s) 2024-04-12T21:51:20.668 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:20 smithi097 ceph-mon[24835]: overall HEALTH_OK 2024-04-12T21:51:20.668 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:20 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:20.668 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:20 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T21:51:20.669 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:20 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:51:21.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:21 smithi097 ceph-mon[24835]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:21.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:21 smithi097 ceph-mon[24835]: Deploying daemon mon.b on smithi114 2024-04-12T21:51:21.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:21 smithi097 ceph-mon[24835]: from='client.? 172.21.15.114:0/4142536587' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-12T21:51:21.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:21 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T21:51:21.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:21 smithi097 ceph-mon[29620]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:21.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:21 smithi097 ceph-mon[29620]: Deploying daemon mon.b on smithi114 2024-04-12T21:51:21.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:21 smithi097 ceph-mon[29620]: from='client.? 172.21.15.114:0/4142536587' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-12T21:51:21.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:21 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T21:51:22.122 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-12T21:51:22.123 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- ceph mon dump -f json 2024-04-12T21:51:23.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:23 smithi097 ceph-mon[24835]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:23.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:23 smithi097 ceph-mon[29620]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:24.209 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:24 smithi114 systemd[1]: Starting Ceph mon.b for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T21:51:25.118 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 podman[25726]: 2024-04-12 21:51:25.01971384 +0000 UTC m=+0.111039332 container create 47a08e70f856e75149a04c8cb8f8c88af766d0777608c1ed8b8a247d3b52cc5b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mon-b, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, distribution-scope=public, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, architecture=x86_64, io.buildah.version=1.19.8, release=754, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-04-12T21:51:25.118 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 podman[25726]: 2024-04-12 21:51:24.926928321 +0000 UTC m=+0.018253842 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-12T21:51:25.154 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-12T21:51:25.154 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":2,"fsid":"80bf7680-f916-11ee-bc8d-c7b262605968","modified":"2024-04-12T21:51:15.295884Z","created":"2024-04-12T21:49:45.534140Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:3300","nonce":0},{"type":"v1","addr":"172.21.15.97:6789","nonce":0}]},"addr":"172.21.15.97:6789/0","public_addr":"172.21.15.97:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:3301","nonce":0},{"type":"v1","addr":"172.21.15.97:6790","nonce":0}]},"addr":"172.21.15.97:6790/0","public_addr":"172.21.15.97:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-12T21:51:25.155 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 2 2024-04-12T21:51:25.382 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 podman[25726]: 2024-04-12 21:51:25.197583628 +0000 UTC m=+0.288909121 container init 47a08e70f856e75149a04c8cb8f8c88af766d0777608c1ed8b8a247d3b52cc5b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mon-b, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, release=754, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, version=8, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., vcs-type=git, distribution-scope=public) 2024-04-12T21:51:25.382 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 podman[25726]: 2024-04-12 21:51:25.201048063 +0000 UTC m=+0.292373557 container start 47a08e70f856e75149a04c8cb8f8c88af766d0777608c1ed8b8a247d3b52cc5b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mon-b, architecture=x86_64, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , name=centos-stream, GIT_CLEAN=True, vendor=Red Hat, Inc., version=8, release=754, distribution-scope=public, com.redhat.component=centos-stream-container, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, io.buildah.version=1.19.8) 2024-04-12T21:51:25.382 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: set uid:gid to 167:167 (ceph:ceph) 2024-04-12T21:51:25.382 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-04-12T21:51:25.382 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: pidfile_write: ignore empty --pid-file 2024-04-12T21:51:25.382 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: load: jerasure load: lrc 2024-04-12T21:51:25.383 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: RocksDB version: 6.15.5 2024-04-12T21:51:25.383 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-04-12T21:51:25.383 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Compile date Apr 18 2022 2024-04-12T21:51:25.383 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: DB SUMMARY 2024-04-12T21:51:25.383 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: DB Session ID: AUIX0N5NMOEREJL8PVR4 2024-04-12T21:51:25.383 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: CURRENT file: CURRENT 2024-04-12T21:51:25.383 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: IDENTITY file: IDENTITY 2024-04-12T21:51:25.383 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-04-12T21:51:25.383 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-04-12T21:51:25.383 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-04-12T21:51:25.383 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.error_if_exists: 0 2024-04-12T21:51:25.383 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.create_if_missing: 0 2024-04-12T21:51:25.384 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.paranoid_checks: 1 2024-04-12T21:51:25.384 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-12T21:51:25.384 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.env: 0x55a1f46c6860 2024-04-12T21:51:25.384 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.fs: Posix File System 2024-04-12T21:51:25.384 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.info_log: 0x55a1f61a7dc0 2024-04-12T21:51:25.384 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.max_file_opening_threads: 16 2024-04-12T21:51:25.384 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.statistics: (nil) 2024-04-12T21:51:25.384 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.use_fsync: 0 2024-04-12T21:51:25.384 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.max_log_file_size: 0 2024-04-12T21:51:25.384 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-12T21:51:25.384 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-12T21:51:25.384 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.keep_log_file_num: 1000 2024-04-12T21:51:25.384 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.recycle_log_file_num: 0 2024-04-12T21:51:25.384 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.allow_fallocate: 1 2024-04-12T21:51:25.384 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.allow_mmap_reads: 0 2024-04-12T21:51:25.385 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.allow_mmap_writes: 0 2024-04-12T21:51:25.385 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.use_direct_reads: 0 2024-04-12T21:51:25.385 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-12T21:51:25.385 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.create_missing_column_families: 0 2024-04-12T21:51:25.385 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.db_log_dir: 2024-04-12T21:51:25.385 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-b/store.db 2024-04-12T21:51:25.385 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-12T21:51:25.385 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-12T21:51:25.385 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-12T21:51:25.385 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-12T21:51:25.385 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-12T21:51:25.385 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-12T21:51:25.387 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.advise_random_on_open: 1 2024-04-12T21:51:25.387 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.db_write_buffer_size: 0 2024-04-12T21:51:25.387 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.write_buffer_manager: 0x55a1f6298240 2024-04-12T21:51:25.387 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-12T21:51:25.387 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-04-12T21:51:25.387 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-12T21:51:25.387 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-12T21:51:25.387 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.rate_limiter: (nil) 2024-04-12T21:51:25.387 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-12T21:51:25.387 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.wal_recovery_mode: 2 2024-04-12T21:51:25.388 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.enable_thread_tracking: 0 2024-04-12T21:51:25.388 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.enable_pipelined_write: 0 2024-04-12T21:51:25.388 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.unordered_write: 0 2024-04-12T21:51:25.388 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-12T21:51:25.388 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-12T21:51:25.388 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-12T21:51:25.388 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-12T21:51:25.388 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.row_cache: None 2024-04-12T21:51:25.388 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.wal_filter: None 2024-04-12T21:51:25.388 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-12T21:51:25.388 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.allow_ingest_behind: 0 2024-04-12T21:51:25.388 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.preserve_deletes: 0 2024-04-12T21:51:25.388 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.two_write_queues: 0 2024-04-12T21:51:25.388 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.manual_wal_flush: 0 2024-04-12T21:51:25.389 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.atomic_flush: 0 2024-04-12T21:51:25.389 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-12T21:51:25.389 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-12T21:51:25.389 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-12T21:51:25.389 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.log_readahead_size: 0 2024-04-12T21:51:25.389 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-12T21:51:25.389 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.best_efforts_recovery: 0 2024-04-12T21:51:25.389 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-12T21:51:25.389 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-12T21:51:25.389 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.allow_data_in_errors: 0 2024-04-12T21:51:25.389 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.db_host_id: __hostname__ 2024-04-12T21:51:25.389 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.max_background_jobs: 2 2024-04-12T21:51:25.389 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.max_background_compactions: -1 2024-04-12T21:51:25.389 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.max_subcompactions: 1 2024-04-12T21:51:25.389 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-12T21:51:25.389 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-12T21:51:25.390 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-12T21:51:25.390 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.max_total_wal_size: 0 2024-04-12T21:51:25.390 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-12T21:51:25.390 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-12T21:51:25.390 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-12T21:51:25.390 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-12T21:51:25.390 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.max_open_files: -1 2024-04-12T21:51:25.390 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.bytes_per_sync: 0 2024-04-12T21:51:25.390 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-12T21:51:25.390 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-12T21:51:25.390 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.compaction_readahead_size: 0 2024-04-12T21:51:25.390 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.max_background_flushes: -1 2024-04-12T21:51:25.390 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Compression algorithms supported: 2024-04-12T21:51:25.390 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-12T21:51:25.390 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: kZSTD supported: 0 2024-04-12T21:51:25.391 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: kXpressCompression supported: 0 2024-04-12T21:51:25.391 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: kLZ4HCCompression supported: 1 2024-04-12T21:51:25.391 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: kLZ4Compression supported: 1 2024-04-12T21:51:25.391 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: kBZip2Compression supported: 0 2024-04-12T21:51:25.391 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: kZlibCompression supported: 1 2024-04-12T21:51:25.391 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: kSnappyCompression supported: 1 2024-04-12T21:51:25.391 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-12T21:51:25.391 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 2024-04-12T21:51:25.391 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-04-12T21:51:25.391 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-12T21:51:25.391 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.merge_operator: 2024-04-12T21:51:25.391 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.compaction_filter: None 2024-04-12T21:51:25.391 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.compaction_filter_factory: None 2024-04-12T21:51:25.392 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.sst_partitioner_factory: None 2024-04-12T21:51:25.392 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-12T21:51:25.392 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-12T21:51:25.392 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55a1f6173d00) 2024-04-12T21:51:25.392 INFO:journalctl@ceph.mon.b.smithi114.stdout: cache_index_and_filter_blocks: 1 2024-04-12T21:51:25.392 INFO:journalctl@ceph.mon.b.smithi114.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-12T21:51:25.392 INFO:journalctl@ceph.mon.b.smithi114.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-12T21:51:25.392 INFO:journalctl@ceph.mon.b.smithi114.stdout: pin_top_level_index_and_filter: 1 2024-04-12T21:51:25.392 INFO:journalctl@ceph.mon.b.smithi114.stdout: index_type: 0 2024-04-12T21:51:25.392 INFO:journalctl@ceph.mon.b.smithi114.stdout: data_block_index_type: 0 2024-04-12T21:51:25.392 INFO:journalctl@ceph.mon.b.smithi114.stdout: index_shortening: 1 2024-04-12T21:51:25.392 INFO:journalctl@ceph.mon.b.smithi114.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-12T21:51:25.392 INFO:journalctl@ceph.mon.b.smithi114.stdout: hash_index_allow_collision: 1 2024-04-12T21:51:25.392 INFO:journalctl@ceph.mon.b.smithi114.stdout: checksum: 1 2024-04-12T21:51:25.393 INFO:journalctl@ceph.mon.b.smithi114.stdout: no_block_cache: 0 2024-04-12T21:51:25.393 INFO:journalctl@ceph.mon.b.smithi114.stdout: block_cache: 0x55a1f61de170 2024-04-12T21:51:25.393 INFO:journalctl@ceph.mon.b.smithi114.stdout: block_cache_name: BinnedLRUCache 2024-04-12T21:51:25.393 INFO:journalctl@ceph.mon.b.smithi114.stdout: block_cache_options: 2024-04-12T21:51:25.393 INFO:journalctl@ceph.mon.b.smithi114.stdout: capacity : 536870912 2024-04-12T21:51:25.393 INFO:journalctl@ceph.mon.b.smithi114.stdout: num_shard_bits : 4 2024-04-12T21:51:25.393 INFO:journalctl@ceph.mon.b.smithi114.stdout: strict_capacity_limit : 0 2024-04-12T21:51:25.393 INFO:journalctl@ceph.mon.b.smithi114.stdout: high_pri_pool_ratio: 0.000 2024-04-12T21:51:25.393 INFO:journalctl@ceph.mon.b.smithi114.stdout: block_cache_compressed: (nil) 2024-04-12T21:51:25.393 INFO:journalctl@ceph.mon.b.smithi114.stdout: persistent_cache: (nil) 2024-04-12T21:51:25.393 INFO:journalctl@ceph.mon.b.smithi114.stdout: block_size: 4096 2024-04-12T21:51:25.393 INFO:journalctl@ceph.mon.b.smithi114.stdout: block_size_deviation: 10 2024-04-12T21:51:25.393 INFO:journalctl@ceph.mon.b.smithi114.stdout: block_restart_interval: 16 2024-04-12T21:51:25.394 INFO:journalctl@ceph.mon.b.smithi114.stdout: index_block_restart_interval: 1 2024-04-12T21:51:25.394 INFO:journalctl@ceph.mon.b.smithi114.stdout: metadata_block_size: 4096 2024-04-12T21:51:25.394 INFO:journalctl@ceph.mon.b.smithi114.stdout: partition_filters: 0 2024-04-12T21:51:25.394 INFO:journalctl@ceph.mon.b.smithi114.stdout: use_delta_encoding: 1 2024-04-12T21:51:25.394 INFO:journalctl@ceph.mon.b.smithi114.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-04-12T21:51:25.394 INFO:journalctl@ceph.mon.b.smithi114.stdout: whole_key_filtering: 1 2024-04-12T21:51:25.394 INFO:journalctl@ceph.mon.b.smithi114.stdout: verify_compression: 0 2024-04-12T21:51:25.394 INFO:journalctl@ceph.mon.b.smithi114.stdout: read_amp_bytes_per_bit: 0 2024-04-12T21:51:25.394 INFO:journalctl@ceph.mon.b.smithi114.stdout: format_version: 4 2024-04-12T21:51:25.394 INFO:journalctl@ceph.mon.b.smithi114.stdout: enable_index_compression: 1 2024-04-12T21:51:25.394 INFO:journalctl@ceph.mon.b.smithi114.stdout: block_align: 0 2024-04-12T21:51:25.394 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.write_buffer_size: 33554432 2024-04-12T21:51:25.394 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.max_write_buffer_number: 2 2024-04-12T21:51:25.394 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.compression: NoCompression 2024-04-12T21:51:25.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.bottommost_compression: Disabled 2024-04-12T21:51:25.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.prefix_extractor: nullptr 2024-04-12T21:51:25.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-12T21:51:25.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.num_levels: 7 2024-04-12T21:51:25.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-12T21:51:25.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-12T21:51:25.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-12T21:51:25.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-12T21:51:25.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-12T21:51:25.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-12T21:51:25.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-12T21:51:25.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-12T21:51:25.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-12T21:51:25.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-12T21:51:25.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-12T21:51:25.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.compression_opts.level: 32767 2024-04-12T21:51:25.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.compression_opts.strategy: 0 2024-04-12T21:51:25.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-12T21:51:25.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-12T21:51:25.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-12T21:51:25.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.compression_opts.enabled: false 2024-04-12T21:51:25.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-12T21:51:25.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-12T21:51:25.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-12T21:51:25.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.target_file_size_base: 67108864 2024-04-12T21:51:25.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-12T21:51:25.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-12T21:51:25.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-12T21:51:25.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-12T21:51:25.396 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-12T21:51:25.397 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-12T21:51:25.397 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-12T21:51:25.397 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-12T21:51:25.397 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-12T21:51:25.397 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-12T21:51:25.397 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-12T21:51:25.397 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-12T21:51:25.397 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-12T21:51:25.397 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.arena_block_size: 4194304 2024-04-12T21:51:25.397 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-12T21:51:25.397 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-12T21:51:25.397 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-04-12T21:51:25.397 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.disable_auto_compactions: 0 2024-04-12T21:51:25.397 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-12T21:51:25.397 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-12T21:51:25.398 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-12T21:51:25.398 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-12T21:51:25.398 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-12T21:51:25.398 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-12T21:51:25.398 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-12T21:51:25.398 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-12T21:51:25.398 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-12T21:51:25.398 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-12T21:51:25.398 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.table_properties_collectors: 2024-04-12T21:51:25.398 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.inplace_update_support: 0 2024-04-12T21:51:25.398 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-12T21:51:25.398 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-12T21:51:25.398 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-12T21:51:25.398 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-12T21:51:25.399 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.bloom_locality: 0 2024-04-12T21:51:25.399 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.max_successive_merges: 0 2024-04-12T21:51:25.399 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-12T21:51:25.399 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.paranoid_file_checks: 0 2024-04-12T21:51:25.399 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.force_consistency_checks: 1 2024-04-12T21:51:25.399 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.report_bg_io_stats: 0 2024-04-12T21:51:25.399 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.ttl: 2592000 2024-04-12T21:51:25.399 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-12T21:51:25.399 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.enable_blob_files: false 2024-04-12T21:51:25.399 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.min_blob_size: 0 2024-04-12T21:51:25.399 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.blob_file_size: 268435456 2024-04-12T21:51:25.399 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-12T21:51:25.399 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-12T21:51:25.399 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-12T21:51:25.399 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-12T21:51:25.400 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 0 2024-04-12T21:51:25.400 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: [db/version_set.cc:4083] Creating manifest 7 2024-04-12T21:51:25.400 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712958685226848, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-12T21:51:25.400 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #4 mode 2 2024-04-12T21:51:25.400 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-04-12T21:51:25.400 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712958685227136, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1670, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1712958685, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "9d0c13eb-c1e3-4135-920f-317c9a8c7906", "db_session_id": "AUIX0N5NMOEREJL8PVR4"}} 2024-04-12T21:51:25.400 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: [db/version_set.cc:4083] Creating manifest 9 2024-04-12T21:51:25.400 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712958685227448, "job": 1, "event": "recovery_finished"} 2024-04-12T21:51:25.400 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-12T21:51:25.400 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x55a1f61c4a80 2024-04-12T21:51:25.400 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: DB pointer 0x55a1f61d4000 2024-04-12T21:51:25.400 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-04-12T21:51:25.400 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-04-12T21:51:25.400 INFO:journalctl@ceph.mon.b.smithi114.stdout: ** DB Stats ** 2024-04-12T21:51:25.400 INFO:journalctl@ceph.mon.b.smithi114.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-12T21:51:25.401 INFO:journalctl@ceph.mon.b.smithi114.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-12T21:51:25.401 INFO:journalctl@ceph.mon.b.smithi114.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-12T21:51:25.401 INFO:journalctl@ceph.mon.b.smithi114.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-12T21:51:25.401 INFO:journalctl@ceph.mon.b.smithi114.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-12T21:51:25.401 INFO:journalctl@ceph.mon.b.smithi114.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-04-12T21:51:25.401 INFO:journalctl@ceph.mon.b.smithi114.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-12T21:51:25.401 INFO:journalctl@ceph.mon.b.smithi114.stdout: 2024-04-12T21:51:25.401 INFO:journalctl@ceph.mon.b.smithi114.stdout: ** Compaction Stats [default] ** 2024-04-12T21:51:25.401 INFO:journalctl@ceph.mon.b.smithi114.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-12T21:51:25.401 INFO:journalctl@ceph.mon.b.smithi114.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-12T21:51:25.401 INFO:journalctl@ceph.mon.b.smithi114.stdout: L0 1/0 1.63 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.7 0.00 0.00 1 0.000 0 0 2024-04-12T21:51:25.401 INFO:journalctl@ceph.mon.b.smithi114.stdout: Sum 1/0 1.63 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.7 0.00 0.00 1 0.000 0 0 2024-04-12T21:51:25.401 INFO:journalctl@ceph.mon.b.smithi114.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.7 0.00 0.00 1 0.000 0 0 2024-04-12T21:51:25.401 INFO:journalctl@ceph.mon.b.smithi114.stdout: 2024-04-12T21:51:25.401 INFO:journalctl@ceph.mon.b.smithi114.stdout: ** Compaction Stats [default] ** 2024-04-12T21:51:25.402 INFO:journalctl@ceph.mon.b.smithi114.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-12T21:51:25.402 INFO:journalctl@ceph.mon.b.smithi114.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-12T21:51:25.402 INFO:journalctl@ceph.mon.b.smithi114.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.7 0.00 0.00 1 0.000 0 0 2024-04-12T21:51:25.402 INFO:journalctl@ceph.mon.b.smithi114.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-12T21:51:25.402 INFO:journalctl@ceph.mon.b.smithi114.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-12T21:51:25.402 INFO:journalctl@ceph.mon.b.smithi114.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-12T21:51:25.402 INFO:journalctl@ceph.mon.b.smithi114.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-12T21:51:25.402 INFO:journalctl@ceph.mon.b.smithi114.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-12T21:51:25.402 INFO:journalctl@ceph.mon.b.smithi114.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-12T21:51:25.402 INFO:journalctl@ceph.mon.b.smithi114.stdout: Cumulative compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-12T21:51:25.402 INFO:journalctl@ceph.mon.b.smithi114.stdout: Interval compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-12T21:51:25.402 INFO:journalctl@ceph.mon.b.smithi114.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-12T21:51:25.402 INFO:journalctl@ceph.mon.b.smithi114.stdout: 2024-04-12T21:51:25.402 INFO:journalctl@ceph.mon.b.smithi114.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-12T21:51:25.403 INFO:journalctl@ceph.mon.b.smithi114.stdout: 2024-04-12T21:51:25.403 INFO:journalctl@ceph.mon.b.smithi114.stdout: ** Compaction Stats [default] ** 2024-04-12T21:51:25.403 INFO:journalctl@ceph.mon.b.smithi114.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-12T21:51:25.403 INFO:journalctl@ceph.mon.b.smithi114.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-12T21:51:25.403 INFO:journalctl@ceph.mon.b.smithi114.stdout: L0 1/0 1.63 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.7 0.00 0.00 1 0.000 0 0 2024-04-12T21:51:25.403 INFO:journalctl@ceph.mon.b.smithi114.stdout: Sum 1/0 1.63 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.7 0.00 0.00 1 0.000 0 0 2024-04-12T21:51:25.403 INFO:journalctl@ceph.mon.b.smithi114.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-04-12T21:51:25.403 INFO:journalctl@ceph.mon.b.smithi114.stdout: 2024-04-12T21:51:25.403 INFO:journalctl@ceph.mon.b.smithi114.stdout: ** Compaction Stats [default] ** 2024-04-12T21:51:25.403 INFO:journalctl@ceph.mon.b.smithi114.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-12T21:51:25.403 INFO:journalctl@ceph.mon.b.smithi114.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-12T21:51:25.403 INFO:journalctl@ceph.mon.b.smithi114.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.7 0.00 0.00 1 0.000 0 0 2024-04-12T21:51:25.403 INFO:journalctl@ceph.mon.b.smithi114.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-12T21:51:25.403 INFO:journalctl@ceph.mon.b.smithi114.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-12T21:51:25.403 INFO:journalctl@ceph.mon.b.smithi114.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-12T21:51:25.404 INFO:journalctl@ceph.mon.b.smithi114.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-12T21:51:25.404 INFO:journalctl@ceph.mon.b.smithi114.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-12T21:51:25.404 INFO:journalctl@ceph.mon.b.smithi114.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-12T21:51:25.404 INFO:journalctl@ceph.mon.b.smithi114.stdout: Cumulative compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-12T21:51:25.404 INFO:journalctl@ceph.mon.b.smithi114.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-12T21:51:25.404 INFO:journalctl@ceph.mon.b.smithi114.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-12T21:51:25.404 INFO:journalctl@ceph.mon.b.smithi114.stdout: 2024-04-12T21:51:25.404 INFO:journalctl@ceph.mon.b.smithi114.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-12T21:51:25.404 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-04-12T21:51:25.404 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: using public_addr v2:172.21.15.114:0/0 -> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-04-12T21:51:25.404 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: starting mon.b rank -1 at public addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] at bind addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 80bf7680-f916-11ee-bc8d-c7b262605968 2024-04-12T21:51:25.404 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: mon.b@-1(???) e0 preinit fsid 80bf7680-f916-11ee-bc8d-c7b262605968 2024-04-12T21:51:25.404 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: mon.b@-1(synchronizing).mds e1 new map 2024-04-12T21:51:25.404 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: mon.b@-1(synchronizing).mds e1 print_map 2024-04-12T21:51:25.404 INFO:journalctl@ceph.mon.b.smithi114.stdout: e1 2024-04-12T21:51:25.405 INFO:journalctl@ceph.mon.b.smithi114.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T21:51:25.405 INFO:journalctl@ceph.mon.b.smithi114.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T21:51:25.405 INFO:journalctl@ceph.mon.b.smithi114.stdout: legacy client fscid: -1 2024-04-12T21:51:25.405 INFO:journalctl@ceph.mon.b.smithi114.stdout: 2024-04-12T21:51:25.405 INFO:journalctl@ceph.mon.b.smithi114.stdout: No filesystems configured 2024-04-12T21:51:25.405 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-12T21:51:25.405 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-12T21:51:25.405 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-12T21:51:25.405 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-12T21:51:25.405 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-12T21:51:25.405 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-12T21:51:25.405 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-12T21:51:25.405 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-12T21:51:25.405 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-12T21:51:25.406 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-12T21:51:25.406 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:25.406 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:25.406 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:25.406 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi114", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:51:25.406 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:25.406 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/139514070' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-12T21:51:25.406 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/139514070' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-12T21:51:25.406 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: osdmap e4: 0 total, 0 up, 0 in 2024-04-12T21:51:25.406 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:25.406 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi097:172.21.15.97=a;smithi097:[v2:172.21.15.97:3301,v1:172.21.15.97:6790]=c;smithi114:172.21.15.114=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:51:25.406 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: Saving service mon spec with placement smithi097:172.21.15.97=a;smithi097:[v2:172.21.15.97:3301,v1:172.21.15.97:6790]=c;smithi114:172.21.15.114=b;count:3 2024-04-12T21:51:25.406 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:25.406 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:51:25.407 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:51:25.407 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:51:25.407 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: Updating smithi114:/etc/ceph/ceph.conf 2024-04-12T21:51:25.407 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-04-12T21:51:25.407 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:25.407 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:25.407 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T21:51:25.407 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:51:25.407 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: Deploying daemon mon.c on smithi097 2024-04-12T21:51:25.407 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:25.407 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: from='client.? 172.21.15.114:0/370944443' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-12T21:51:25.407 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:25.407 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-12T21:51:25.407 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: mon.a calling monitor election 2024-04-12T21:51:25.407 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T21:51:25.408 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:25.408 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T21:51:25.408 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T21:51:25.408 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: mon.c calling monitor election 2024-04-12T21:51:25.408 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:25.408 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T21:51:25.408 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T21:51:25.408 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T21:51:25.408 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-12T21:51:25.408 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: monmap e2: 2 mons at {a=[v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0],c=[v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0]} 2024-04-12T21:51:25.408 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: fsmap 2024-04-12T21:51:25.408 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: osdmap e4: 0 total, 0 up, 0 in 2024-04-12T21:51:25.408 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: mgrmap e13: y(active, since 37s) 2024-04-12T21:51:25.408 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: overall HEALTH_OK 2024-04-12T21:51:25.408 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:25.409 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T21:51:25.409 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:51:25.409 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:25.409 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: Deploying daemon mon.b on smithi114 2024-04-12T21:51:25.409 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: from='client.? 172.21.15.114:0/4142536587' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-12T21:51:25.409 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T21:51:25.409 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:25.409 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-12T21:51:25.409 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 ceph-mon[25759]: mon.b@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-04-12T21:51:25.409 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 bash[25726]: 47a08e70f856e75149a04c8cb8f8c88af766d0777608c1ed8b8a247d3b52cc5b 2024-04-12T21:51:25.409 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:25 smithi114 systemd[1]: Started Ceph mon.b for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T21:51:26.795 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-12T21:51:26.796 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- ceph mon dump -f json 2024-04-12T21:51:30.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:30 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-12T21:51:30.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:30 smithi097 ceph-mon[29620]: mon.a calling monitor election 2024-04-12T21:51:30.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:30 smithi097 ceph-mon[29620]: mon.c calling monitor election 2024-04-12T21:51:30.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:30 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:51:30.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:30 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T21:51:30.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:30 smithi097 ceph-mon[29620]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:30.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:30 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:51:30.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:30 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:51:30.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:30 smithi097 ceph-mon[29620]: mon.b calling monitor election 2024-04-12T21:51:30.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:30 smithi097 ceph-mon[29620]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:30.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:30 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:51:30.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:30 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:51:30.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:30 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:51:30.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:30 smithi097 ceph-mon[29620]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-12T21:51:30.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:30 smithi097 ceph-mon[29620]: monmap e3: 3 mons at {a=[v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0],b=[v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0],c=[v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0]} 2024-04-12T21:51:30.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:30 smithi097 ceph-mon[29620]: fsmap 2024-04-12T21:51:30.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:30 smithi097 ceph-mon[29620]: osdmap e4: 0 total, 0 up, 0 in 2024-04-12T21:51:30.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:30 smithi097 ceph-mon[29620]: mgrmap e13: y(active, since 47s) 2024-04-12T21:51:30.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:30 smithi097 ceph-mon[29620]: overall HEALTH_OK 2024-04-12T21:51:30.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:30 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:30.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:30 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:51:30.663 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:30 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:51:30.663 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:30 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:51:30.663 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:30 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-12T21:51:30.663 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:30 smithi097 ceph-mon[24835]: mon.a calling monitor election 2024-04-12T21:51:30.663 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:30 smithi097 ceph-mon[24835]: mon.c calling monitor election 2024-04-12T21:51:30.663 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:30 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:51:30.663 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:30 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T21:51:30.663 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:30 smithi097 ceph-mon[24835]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:30.663 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:30 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:51:30.663 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:30 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:51:30.663 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:30 smithi097 ceph-mon[24835]: mon.b calling monitor election 2024-04-12T21:51:30.664 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:30 smithi097 ceph-mon[24835]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:30.664 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:30 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:51:30.664 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:30 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:51:30.664 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:30 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:51:30.664 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:30 smithi097 ceph-mon[24835]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-12T21:51:30.664 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:30 smithi097 ceph-mon[24835]: monmap e3: 3 mons at {a=[v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0],b=[v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0],c=[v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0]} 2024-04-12T21:51:30.664 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:30 smithi097 ceph-mon[24835]: fsmap 2024-04-12T21:51:30.664 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:30 smithi097 ceph-mon[24835]: osdmap e4: 0 total, 0 up, 0 in 2024-04-12T21:51:30.664 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:30 smithi097 ceph-mon[24835]: mgrmap e13: y(active, since 47s) 2024-04-12T21:51:30.664 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:30 smithi097 ceph-mon[24835]: overall HEALTH_OK 2024-04-12T21:51:30.664 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:30 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:30.664 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:30 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:51:30.664 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:30 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:51:30.665 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:30 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:51:30.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:30 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-12T21:51:30.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:30 smithi114 ceph-mon[25759]: mon.a calling monitor election 2024-04-12T21:51:30.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:30 smithi114 ceph-mon[25759]: mon.c calling monitor election 2024-04-12T21:51:30.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:30 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:51:30.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:30 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T21:51:30.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:30 smithi114 ceph-mon[25759]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:30.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:30 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:51:30.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:30 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:51:30.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:30 smithi114 ceph-mon[25759]: mon.b calling monitor election 2024-04-12T21:51:30.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:30 smithi114 ceph-mon[25759]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:30.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:30 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:51:30.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:30 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:51:30.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:30 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:51:30.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:30 smithi114 ceph-mon[25759]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-12T21:51:30.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:30 smithi114 ceph-mon[25759]: monmap e3: 3 mons at {a=[v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0],b=[v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0],c=[v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0]} 2024-04-12T21:51:30.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:30 smithi114 ceph-mon[25759]: fsmap 2024-04-12T21:51:30.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:30 smithi114 ceph-mon[25759]: osdmap e4: 0 total, 0 up, 0 in 2024-04-12T21:51:30.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:30 smithi114 ceph-mon[25759]: mgrmap e13: y(active, since 47s) 2024-04-12T21:51:30.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:30 smithi114 ceph-mon[25759]: overall HEALTH_OK 2024-04-12T21:51:30.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:30 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:30.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:30 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:51:30.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:30 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:51:30.746 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:30 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:51:31.466 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:31 smithi114 ceph-mon[25759]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:31.467 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:31 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:31.467 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:31 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:51:31.467 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:31 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:31.546 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:31 smithi097 ceph-mon[24835]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:31.546 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:31 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:31.546 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:31 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:51:31.546 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:31 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:31.547 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:31 smithi097 ceph-mon[29620]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:31.547 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:31 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:31.547 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:31 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:51:31.547 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:31 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:31.782 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-12T21:51:31.782 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":3,"fsid":"80bf7680-f916-11ee-bc8d-c7b262605968","modified":"2024-04-12T21:51:25.246349Z","created":"2024-04-12T21:49:45.534140Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:3300","nonce":0},{"type":"v1","addr":"172.21.15.97:6789","nonce":0}]},"addr":"172.21.15.97:6789/0","public_addr":"172.21.15.97:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:3301","nonce":0},{"type":"v1","addr":"172.21.15.97:6790","nonce":0}]},"addr":"172.21.15.97:6790/0","public_addr":"172.21.15.97:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-04-12T21:51:31.782 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 3 2024-04-12T21:51:32.333 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:32 smithi114 ceph-mon[25759]: Updating smithi114:/etc/ceph/ceph.conf 2024-04-12T21:51:32.333 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:32 smithi114 ceph-mon[25759]: Updating smithi097:/etc/ceph/ceph.conf 2024-04-12T21:51:32.334 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:32 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:32.334 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:32 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:32.334 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:32 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:32.334 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:32 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T21:51:32.334 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:32 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T21:51:32.334 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:32 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:51:32.335 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:32 smithi114 ceph-mon[25759]: from='client.? 172.21.15.114:0/3597683826' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-12T21:51:32.355 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-12T21:51:32.356 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- ceph config generate-minimal-conf 2024-04-12T21:51:32.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:32 smithi097 ceph-mon[29620]: Updating smithi114:/etc/ceph/ceph.conf 2024-04-12T21:51:32.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:32 smithi097 ceph-mon[29620]: Updating smithi097:/etc/ceph/ceph.conf 2024-04-12T21:51:32.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:32 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:32.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:32 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:32.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:32 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:32.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:32 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T21:51:32.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:32 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T21:51:32.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:32 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:51:32.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:32 smithi097 ceph-mon[29620]: from='client.? 172.21.15.114:0/3597683826' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-12T21:51:32.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:32 smithi097 ceph-mon[24835]: Updating smithi114:/etc/ceph/ceph.conf 2024-04-12T21:51:32.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:32 smithi097 ceph-mon[24835]: Updating smithi097:/etc/ceph/ceph.conf 2024-04-12T21:51:32.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:32 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:32.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:32 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:32.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:32 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:32.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:32 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T21:51:32.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:32 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T21:51:32.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:32 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:51:32.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:32 smithi097 ceph-mon[24835]: from='client.? 172.21.15.114:0/3597683826' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-12T21:51:33.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:33 smithi097 ceph-mon[29620]: Reconfiguring mon.a (unknown last config time)... 2024-04-12T21:51:33.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:33 smithi097 ceph-mon[29620]: Reconfiguring daemon mon.a on smithi097 2024-04-12T21:51:33.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:33 smithi097 ceph-mon[29620]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:33.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:33 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:33.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:33 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T21:51:33.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:33 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T21:51:33.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:33 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:51:33.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:33 smithi097 ceph-mon[24835]: Reconfiguring mon.a (unknown last config time)... 2024-04-12T21:51:33.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:33 smithi097 ceph-mon[24835]: Reconfiguring daemon mon.a on smithi097 2024-04-12T21:51:33.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:33 smithi097 ceph-mon[24835]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:33.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:33 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:33.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:33 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T21:51:33.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:33 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T21:51:33.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:33 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:51:33.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:33 smithi114 ceph-mon[25759]: Reconfiguring mon.a (unknown last config time)... 2024-04-12T21:51:33.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:33 smithi114 ceph-mon[25759]: Reconfiguring daemon mon.a on smithi097 2024-04-12T21:51:33.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:33 smithi114 ceph-mon[25759]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:33.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:33 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:33.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:33 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T21:51:33.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:33 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T21:51:33.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:33 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:51:34.559 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:34 smithi097 ceph-mon[24835]: Reconfiguring mon.c (monmap changed)... 2024-04-12T21:51:34.559 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:34 smithi097 ceph-mon[24835]: Reconfiguring daemon mon.c on smithi097 2024-04-12T21:51:34.559 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:34 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:34.559 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:34 smithi097 ceph-mon[29620]: Reconfiguring mon.c (monmap changed)... 2024-04-12T21:51:34.559 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:34 smithi097 ceph-mon[29620]: Reconfiguring daemon mon.c on smithi097 2024-04-12T21:51:34.559 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:34 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:34.625 INFO:teuthology.orchestra.run.smithi097.stdout:# minimal ceph.conf for 80bf7680-f916-11ee-bc8d-c7b262605968 2024-04-12T21:51:34.626 INFO:teuthology.orchestra.run.smithi097.stdout:[global] 2024-04-12T21:51:34.626 INFO:teuthology.orchestra.run.smithi097.stdout: fsid = 80bf7680-f916-11ee-bc8d-c7b262605968 2024-04-12T21:51:34.626 INFO:teuthology.orchestra.run.smithi097.stdout: mon_host = [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] 2024-04-12T21:51:34.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:34 smithi114 ceph-mon[25759]: Reconfiguring mon.c (monmap changed)... 2024-04-12T21:51:34.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:34 smithi114 ceph-mon[25759]: Reconfiguring daemon mon.c on smithi097 2024-04-12T21:51:34.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:34 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:35.657 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:35 smithi097 ceph-mon[24835]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:35.657 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:35 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/2312564390' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:51:35.657 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:35 smithi097 ceph-mon[29620]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:35.658 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:35 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/2312564390' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:51:35.686 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-12T21:51:35.686 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-12T21:51:35.687 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/ceph/ceph.conf 2024-04-12T21:51:35.720 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-12T21:51:35.721 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-12T21:51:35.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:35 smithi114 ceph-mon[25759]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:35.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:35 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/2312564390' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:51:35.788 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-04-12T21:51:35.788 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/ceph/ceph.conf 2024-04-12T21:51:35.820 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-04-12T21:51:35.820 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-12T21:51:35.889 INFO:tasks.cephadm:Adding mgr.y on smithi097 2024-04-12T21:51:35.891 INFO:tasks.cephadm:Adding mgr.x on smithi114 2024-04-12T21:51:35.892 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- ceph orch apply mgr '2;smithi097=y;smithi114=x' 2024-04-12T21:51:37.417 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:37 smithi114 ceph-mon[25759]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:37.417 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:37 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:37.417 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:37 smithi114 ceph-mon[25759]: Reconfiguring mon.b (monmap changed)... 2024-04-12T21:51:37.417 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:37 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T21:51:37.417 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:37 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T21:51:37.417 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:37 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:51:37.417 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:37 smithi114 ceph-mon[25759]: Reconfiguring daemon mon.b on smithi114 2024-04-12T21:51:37.655 INFO:teuthology.orchestra.run.smithi114.stdout:Scheduled mgr update... 2024-04-12T21:51:37.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:37 smithi097 ceph-mon[24835]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:37.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:37 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:37.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:37 smithi097 ceph-mon[24835]: Reconfiguring mon.b (monmap changed)... 2024-04-12T21:51:37.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:37 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T21:51:37.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:37 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T21:51:37.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:37 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:51:37.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:37 smithi097 ceph-mon[24835]: Reconfiguring daemon mon.b on smithi114 2024-04-12T21:51:37.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:37 smithi097 ceph-mon[29620]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:37.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:37 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:37.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:37 smithi097 ceph-mon[29620]: Reconfiguring mon.b (monmap changed)... 2024-04-12T21:51:37.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:37 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T21:51:37.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:37 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T21:51:37.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:37 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:51:37.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:37 smithi097 ceph-mon[29620]: Reconfiguring daemon mon.b on smithi114 2024-04-12T21:51:38.271 DEBUG:teuthology.orchestra.run.smithi114:mgr.x> sudo journalctl -f -n 0 -u ceph-80bf7680-f916-11ee-bc8d-c7b262605968@mgr.x.service 2024-04-12T21:51:38.273 INFO:tasks.cephadm:Deploying OSDs... 2024-04-12T21:51:38.274 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-12T21:51:38.274 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/scratch_devs of=/dev/stdout 2024-04-12T21:51:38.293 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-12T21:51:38.294 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_1 2024-04-12T21:51:38.351 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-12T21:51:38.351 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-12T21:51:38.351 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-12T21:51:38.351 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-12T21:51:38.351 INFO:teuthology.orchestra.run.smithi097.stdout:Context: system_u:object_r:device_t:s0 2024-04-12T21:51:38.351 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2024-04-12 21:51:14.787302556 +0000 2024-04-12T21:51:38.352 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2024-04-12 21:48:47.579428485 +0000 2024-04-12T21:51:38.352 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2024-04-12 21:48:47.579428485 +0000 2024-04-12T21:51:38.352 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: 2024-04-12 21:48:47.579428485 +0000 2024-04-12T21:51:38.352 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-12T21:51:38.421 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2024-04-12T21:51:38.422 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2024-04-12T21:51:38.422 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000286461 s, 1.8 MB/s 2024-04-12T21:51:38.423 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-12T21:51:38.481 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_2 2024-04-12T21:51:38.539 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-12T21:51:38.539 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-12T21:51:38.539 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-12T21:51:38.539 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-12T21:51:38.539 INFO:teuthology.orchestra.run.smithi097.stdout:Context: system_u:object_r:device_t:s0 2024-04-12T21:51:38.539 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2024-04-12 21:51:14.787302556 +0000 2024-04-12T21:51:38.539 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2024-04-12 21:48:47.922422986 +0000 2024-04-12T21:51:38.539 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2024-04-12 21:48:47.922422986 +0000 2024-04-12T21:51:38.539 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: 2024-04-12 21:48:47.922422986 +0000 2024-04-12T21:51:38.540 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-12T21:51:38.607 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2024-04-12T21:51:38.607 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2024-04-12T21:51:38.607 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000173244 s, 3.0 MB/s 2024-04-12T21:51:38.608 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-12T21:51:38.666 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_3 2024-04-12T21:51:38.722 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-12T21:51:38.722 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-12T21:51:38.722 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-04-12T21:51:38.722 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-12T21:51:38.722 INFO:teuthology.orchestra.run.smithi097.stdout:Context: system_u:object_r:device_t:s0 2024-04-12T21:51:38.723 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2024-04-12 21:51:14.788302540 +0000 2024-04-12T21:51:38.723 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2024-04-12 21:48:48.276417311 +0000 2024-04-12T21:51:38.723 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2024-04-12 21:48:48.276417311 +0000 2024-04-12T21:51:38.723 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: 2024-04-12 21:48:48.276417311 +0000 2024-04-12T21:51:38.723 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-12T21:51:38.786 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:38 smithi097 ceph-mon[24835]: from='client.24110 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi097=y;smithi114=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:51:38.787 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:38 smithi097 ceph-mon[24835]: Saving service mgr spec with placement smithi097=y;smithi114=x;count:2 2024-04-12T21:51:38.787 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:38 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:38.787 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:38 smithi097 ceph-mon[24835]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:38.787 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:38 smithi097 ceph-mon[29620]: from='client.24110 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi097=y;smithi114=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:51:38.787 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:38 smithi097 ceph-mon[29620]: Saving service mgr spec with placement smithi097=y;smithi114=x;count:2 2024-04-12T21:51:38.787 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:38 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:38.787 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:38 smithi097 ceph-mon[29620]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:38.789 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2024-04-12T21:51:38.789 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2024-04-12T21:51:38.789 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000189804 s, 2.7 MB/s 2024-04-12T21:51:38.790 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-12T21:51:38.846 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_4 2024-04-12T21:51:38.901 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-12T21:51:38.901 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-12T21:51:38.902 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-04-12T21:51:38.902 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-12T21:51:38.902 INFO:teuthology.orchestra.run.smithi097.stdout:Context: system_u:object_r:device_t:s0 2024-04-12T21:51:38.902 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2024-04-12 21:51:14.788302540 +0000 2024-04-12T21:51:38.902 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2024-04-12 21:48:48.618411829 +0000 2024-04-12T21:51:38.902 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2024-04-12 21:48:48.618411829 +0000 2024-04-12T21:51:38.902 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: 2024-04-12 21:48:48.618411829 +0000 2024-04-12T21:51:38.902 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-12T21:51:38.968 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2024-04-12T21:51:38.968 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2024-04-12T21:51:38.968 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000295884 s, 1.7 MB/s 2024-04-12T21:51:38.969 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-12T21:51:38.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:38 smithi114 ceph-mon[25759]: from='client.24110 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi097=y;smithi114=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:51:38.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:38 smithi114 ceph-mon[25759]: Saving service mgr spec with placement smithi097=y;smithi114=x;count:2 2024-04-12T21:51:38.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:38 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:38.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:38 smithi114 ceph-mon[25759]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:39.025 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-04-12T21:51:39.025 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/scratch_devs of=/dev/stdout 2024-04-12T21:51:39.041 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-12T21:51:39.041 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_1 2024-04-12T21:51:39.096 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-12T21:51:39.096 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-12T21:51:39.096 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-12T21:51:39.096 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-12T21:51:39.096 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-04-12T21:51:39.096 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-04-12 21:51:24.040421678 +0000 2024-04-12T21:51:39.096 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-04-12 21:48:47.583199361 +0000 2024-04-12T21:51:39.096 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-04-12 21:48:47.583199361 +0000 2024-04-12T21:51:39.096 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-04-12 21:48:47.583199361 +0000 2024-04-12T21:51:39.097 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-12T21:51:39.162 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-04-12T21:51:39.162 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-04-12T21:51:39.162 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000220839 s, 2.3 MB/s 2024-04-12T21:51:39.163 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-12T21:51:39.218 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_2 2024-04-12T21:51:39.274 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-12T21:51:39.275 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-12T21:51:39.275 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-04-12T21:51:39.275 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-12T21:51:39.275 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-04-12T21:51:39.275 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-04-12 21:51:24.041421656 +0000 2024-04-12T21:51:39.275 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-04-12 21:48:47.934191586 +0000 2024-04-12T21:51:39.275 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-04-12 21:48:47.934191586 +0000 2024-04-12T21:51:39.275 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-04-12 21:48:47.934191586 +0000 2024-04-12T21:51:39.275 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-12T21:51:39.343 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-04-12T21:51:39.343 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-04-12T21:51:39.343 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000220198 s, 2.3 MB/s 2024-04-12T21:51:39.344 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-12T21:51:39.399 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_3 2024-04-12T21:51:39.454 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-12T21:51:39.455 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-12T21:51:39.455 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-04-12T21:51:39.455 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-12T21:51:39.455 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-04-12T21:51:39.455 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-04-12 21:51:24.041421656 +0000 2024-04-12T21:51:39.455 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-04-12 21:48:48.278183966 +0000 2024-04-12T21:51:39.455 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-04-12 21:48:48.278183966 +0000 2024-04-12T21:51:39.455 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-04-12 21:48:48.278183966 +0000 2024-04-12T21:51:39.455 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-12T21:51:39.520 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-04-12T21:51:39.520 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-04-12T21:51:39.520 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000174284 s, 2.9 MB/s 2024-04-12T21:51:39.521 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-12T21:51:39.577 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_4 2024-04-12T21:51:39.632 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-12T21:51:39.632 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-12T21:51:39.633 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-12T21:51:39.633 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-12T21:51:39.633 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-04-12T21:51:39.633 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-04-12 21:51:24.041421656 +0000 2024-04-12T21:51:39.633 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-04-12 21:48:48.617176457 +0000 2024-04-12T21:51:39.633 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-04-12 21:48:48.617176457 +0000 2024-04-12T21:51:39.633 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-04-12 21:48:48.617176457 +0000 2024-04-12T21:51:39.633 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-12T21:51:39.698 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-04-12T21:51:39.698 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-04-12T21:51:39.698 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000211235 s, 2.4 MB/s 2024-04-12T21:51:39.699 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-12T21:51:39.755 INFO:tasks.cephadm:Deploying osd.0 on smithi097 with /dev/vg_nvme/lv_4... 2024-04-12T21:51:39.755 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-12T21:51:40.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:40 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:40.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:40 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:51:40.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:40 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:51:40.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:40 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:51:40.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:40 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:40.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:40 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T21:51:40.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:40 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-12T21:51:40.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:40 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T21:51:40.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:40 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:51:40.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:40 smithi114 ceph-mon[25759]: Deploying daemon mgr.x on smithi114 2024-04-12T21:51:40.293 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:40 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:40.293 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:40 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:51:40.293 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:40 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:51:40.293 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:40 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:51:40.293 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:40 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:40.293 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:40 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T21:51:40.293 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:40 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-12T21:51:40.294 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:40 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T21:51:40.294 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:40 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:51:40.294 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:40 smithi097 ceph-mon[29620]: Deploying daemon mgr.x on smithi114 2024-04-12T21:51:40.294 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:40 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:40.294 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:40 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:51:40.294 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:40 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:51:40.294 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:40 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:51:40.294 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:40 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:40.294 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:40 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T21:51:40.294 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:40 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-12T21:51:40.295 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:40 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T21:51:40.295 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:40 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:51:40.295 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:40 smithi097 ceph-mon[24835]: Deploying daemon mgr.x on smithi114 2024-04-12T21:51:40.921 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:51:40 smithi114 systemd[1]: Starting Ceph mgr.x for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T21:51:41.175 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:41 smithi114 ceph-mon[25759]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:41.176 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:51:40 smithi114 podman[26924]: 2024-04-12 21:51:40.822449091 +0000 UTC m=+0.017993503 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-12T21:51:41.176 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:51:40 smithi114 podman[26924]: 2024-04-12 21:51:40.92420327 +0000 UTC m=+0.119747721 container create 670fe6214c3bb60c4e535b275712d59e327e9aae2e04f18edd9c5dfbb8bb0860 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, GIT_BRANCH=HEAD, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, ceph=True, name=centos-stream) 2024-04-12T21:51:41.176 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:51:41 smithi114 podman[26924]: 2024-04-12 21:51:41.151438068 +0000 UTC m=+0.346982474 container init 670fe6214c3bb60c4e535b275712d59e327e9aae2e04f18edd9c5dfbb8bb0860 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, version=8, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, release=754, name=centos-stream, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, vcs-type=git, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-04-12T21:51:41.176 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:51:41 smithi114 podman[26924]: 2024-04-12 21:51:41.154030191 +0000 UTC m=+0.349574598 container start 670fe6214c3bb60c4e535b275712d59e327e9aae2e04f18edd9c5dfbb8bb0860 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, ceph=True, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, version=8, io.k8s.display-name=CentOS Stream 8, name=centos-stream) 2024-04-12T21:51:41.176 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:51:41 smithi114 ceph-mgr[26943]: -- 172.21.15.114:0/35066703 <== mon.2 v2:172.21.15.114:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e912f56340 con 0x55e912f2f000 2024-04-12T21:51:41.356 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:41 smithi097 ceph-mon[24835]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:41.356 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:41 smithi097 ceph-mon[29620]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:41.493 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:51:41 smithi114 bash[26924]: 670fe6214c3bb60c4e535b275712d59e327e9aae2e04f18edd9c5dfbb8bb0860 2024-04-12T21:51:41.494 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:51:41 smithi114 systemd[1]: Started Ceph mgr.x for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T21:51:41.825 INFO:teuthology.orchestra.run.smithi097.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-12T21:51:41.826 INFO:teuthology.orchestra.run.smithi097.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-12T21:51:41.826 INFO:teuthology.orchestra.run.smithi097.stderr: stderr: 10+0 records in 2024-04-12T21:51:41.826 INFO:teuthology.orchestra.run.smithi097.stderr:10+0 records out 2024-04-12T21:51:41.826 INFO:teuthology.orchestra.run.smithi097.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.012437 s, 843 MB/s 2024-04-12T21:51:41.826 INFO:teuthology.orchestra.run.smithi097.stderr:--> Zapping successful for: 2024-04-12T21:51:41.960 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:51:41 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: 2024-04-12T21:51:41.680+0000 7f95817ca000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-12T21:51:41.960 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:51:41 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: 2024-04-12T21:51:41.769+0000 7f95817ca000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-12T21:51:42.471 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:51:42 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: 2024-04-12T21:51:42.382+0000 7f95817ca000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-12T21:51:42.471 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:51:42 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: 2024-04-12T21:51:42.469+0000 7f95817ca000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-12T21:51:42.472 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:42 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:42.472 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:42 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:51:42.472 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:42 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:51:42.472 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:42 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:51:42.472 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:42 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:42.472 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:42 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:42.472 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:42 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T21:51:42.472 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:42 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T21:51:42.472 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:42 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:51:42.541 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:42 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:42.541 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:42 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:51:42.541 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:42 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:51:42.541 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:42 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:51:42.541 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:42 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:42.541 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:42 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:42.541 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:42 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T21:51:42.541 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:42 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T21:51:42.541 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:42 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:51:42.542 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:42 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:42.542 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:42 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:51:42.542 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:42 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:51:42.542 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:42 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:51:42.542 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:42 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:42.542 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:42 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:42.542 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:42 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T21:51:42.542 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:42 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T21:51:42.542 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:42 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:51:42.562 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- ceph orch daemon add osd smithi097:vg_nvme/lv_4 2024-04-12T21:51:42.743 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:51:42 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: 2024-04-12T21:51:42.552+0000 7f95817ca000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-12T21:51:42.744 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:51:42 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: 2024-04-12T21:51:42.698+0000 7f95817ca000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-12T21:51:43.132 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:51:42 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: 2024-04-12T21:51:42.778+0000 7f95817ca000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-12T21:51:43.132 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:51:42 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: 2024-04-12T21:51:42.887+0000 7f95817ca000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-12T21:51:43.132 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:51:42 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: 2024-04-12T21:51:42.968+0000 7f95817ca000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-12T21:51:43.132 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:51:43 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: 2024-04-12T21:51:43.130+0000 7f95817ca000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-12T21:51:43.493 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:51:43 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: 2024-04-12T21:51:43.214+0000 7f95817ca000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-12T21:51:43.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:43 smithi114 ceph-mon[25759]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:43.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:43 smithi114 ceph-mon[25759]: Reconfiguring mgr.y (unknown last config time)... 2024-04-12T21:51:43.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:43 smithi114 ceph-mon[25759]: Reconfiguring daemon mgr.y on smithi097 2024-04-12T21:51:43.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:43 smithi097 ceph-mon[29620]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:43.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:43 smithi097 ceph-mon[29620]: Reconfiguring mgr.y (unknown last config time)... 2024-04-12T21:51:43.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:43 smithi097 ceph-mon[29620]: Reconfiguring daemon mgr.y on smithi097 2024-04-12T21:51:43.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:43 smithi097 ceph-mon[24835]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:43.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:43 smithi097 ceph-mon[24835]: Reconfiguring mgr.y (unknown last config time)... 2024-04-12T21:51:43.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:43 smithi097 ceph-mon[24835]: Reconfiguring daemon mgr.y on smithi097 2024-04-12T21:51:44.493 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:51:44 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: 2024-04-12T21:51:44.236+0000 7f95817ca000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-12T21:51:44.494 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:51:44 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: 2024-04-12T21:51:44.363+0000 7f95817ca000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-12T21:51:44.881 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:51:44 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: 2024-04-12T21:51:44.550+0000 7f95817ca000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-12T21:51:44.881 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:51:44 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: 2024-04-12T21:51:44.636+0000 7f95817ca000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-12T21:51:44.881 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:51:44 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: 2024-04-12T21:51:44.769+0000 7f95817ca000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-12T21:51:44.983 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:44 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:44.983 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:44 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:51:44.983 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:44 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:51:44.983 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:44 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:51:44.983 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:44 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:44.983 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:44 smithi097 ceph-mon[29620]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:44.983 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:44 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T21:51:44.983 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:44 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:44.983 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:44 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T21:51:44.983 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:44 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:44.984 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:44 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:51:44.984 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:44 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:51:44.984 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:44 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:51:44.984 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:44 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:44.984 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:44 smithi097 ceph-mon[24835]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:44.984 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:44 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T21:51:44.984 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:44 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:44.984 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:44 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T21:51:45.243 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:51:45 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: 2024-04-12T21:51:45.086+0000 7f95817ca000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-12T21:51:45.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:44 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:45.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:44 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:51:45.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:44 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:51:45.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:44 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:51:45.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:44 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:45.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:44 smithi114 ceph-mon[25759]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:45.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:44 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T21:51:45.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:44 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:45.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:44 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T21:51:45.559 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:51:45 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: 2024-04-12T21:51:45.303+0000 7f95817ca000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-12T21:51:45.559 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:51:45 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: 2024-04-12T21:51:45.387+0000 7f95817ca000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-12T21:51:45.559 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:51:45 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: 2024-04-12T21:51:45.557+0000 7f95817ca000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-12T21:51:46.043 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:45 smithi097 ceph-mon[24835]: from='client.24124 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:51:46.043 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:45 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:51:46.043 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:45 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:51:46.043 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:45 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:51:46.043 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:45 smithi097 ceph-mon[29620]: from='client.24124 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:51:46.043 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:45 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:51:46.043 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:45 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:51:46.043 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:45 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:51:46.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:45 smithi114 ceph-mon[25759]: from='client.24124 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:51:46.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:45 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:51:46.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:45 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:51:46.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:45 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:51:46.743 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:51:46 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: 2024-04-12T21:51:46.490+0000 7f95817ca000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-12T21:51:46.940 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:46 smithi097 ceph-mon[24835]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:46.940 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:46 smithi097 ceph-mon[29620]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:47.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:46 smithi114 ceph-mon[25759]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:47.279 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:51:46 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: 2024-04-12T21:51:46.783+0000 7f95817ca000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-12T21:51:47.279 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:51:46 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: 2024-04-12T21:51:46.875+0000 7f95817ca000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-12T21:51:47.281 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:51:47 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: 2024-04-12T21:51:47.027+0000 7f95817ca000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-12T21:51:48.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:47 smithi097 ceph-mon[29620]: Standby manager daemon x started 2024-04-12T21:51:48.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:47 smithi097 ceph-mon[29620]: from='mgr.? 172.21.15.114:0/1375690756' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-12T21:51:48.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:47 smithi097 ceph-mon[29620]: from='mgr.? 172.21.15.114:0/1375690756' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-12T21:51:48.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:47 smithi097 ceph-mon[29620]: from='mgr.? 172.21.15.114:0/1375690756' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-12T21:51:48.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:47 smithi097 ceph-mon[29620]: from='mgr.? 172.21.15.114:0/1375690756' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-12T21:51:48.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:47 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/188550696' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "315f320f-95fd-4b18-ab6a-417ba7651667"}]: dispatch 2024-04-12T21:51:48.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:47 smithi097 ceph-mon[29620]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "315f320f-95fd-4b18-ab6a-417ba7651667"}]: dispatch 2024-04-12T21:51:48.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:47 smithi097 ceph-mon[29620]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "315f320f-95fd-4b18-ab6a-417ba7651667"}]': finished 2024-04-12T21:51:48.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:47 smithi097 ceph-mon[29620]: osdmap e5: 1 total, 0 up, 1 in 2024-04-12T21:51:48.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:47 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T21:51:48.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:47 smithi097 ceph-mon[24835]: Standby manager daemon x started 2024-04-12T21:51:48.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:47 smithi097 ceph-mon[24835]: from='mgr.? 172.21.15.114:0/1375690756' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-12T21:51:48.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:47 smithi097 ceph-mon[24835]: from='mgr.? 172.21.15.114:0/1375690756' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-12T21:51:48.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:47 smithi097 ceph-mon[24835]: from='mgr.? 172.21.15.114:0/1375690756' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-12T21:51:48.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:47 smithi097 ceph-mon[24835]: from='mgr.? 172.21.15.114:0/1375690756' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-12T21:51:48.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:47 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/188550696' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "315f320f-95fd-4b18-ab6a-417ba7651667"}]: dispatch 2024-04-12T21:51:48.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:47 smithi097 ceph-mon[24835]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "315f320f-95fd-4b18-ab6a-417ba7651667"}]: dispatch 2024-04-12T21:51:48.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:47 smithi097 ceph-mon[24835]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "315f320f-95fd-4b18-ab6a-417ba7651667"}]': finished 2024-04-12T21:51:48.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:47 smithi097 ceph-mon[24835]: osdmap e5: 1 total, 0 up, 1 in 2024-04-12T21:51:48.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:47 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T21:51:48.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:47 smithi114 ceph-mon[25759]: Standby manager daemon x started 2024-04-12T21:51:48.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:47 smithi114 ceph-mon[25759]: from='mgr.? 172.21.15.114:0/1375690756' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-12T21:51:48.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:47 smithi114 ceph-mon[25759]: from='mgr.? 172.21.15.114:0/1375690756' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-12T21:51:48.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:47 smithi114 ceph-mon[25759]: from='mgr.? 172.21.15.114:0/1375690756' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-12T21:51:48.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:47 smithi114 ceph-mon[25759]: from='mgr.? 172.21.15.114:0/1375690756' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-12T21:51:48.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:47 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/188550696' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "315f320f-95fd-4b18-ab6a-417ba7651667"}]: dispatch 2024-04-12T21:51:48.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:47 smithi114 ceph-mon[25759]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "315f320f-95fd-4b18-ab6a-417ba7651667"}]: dispatch 2024-04-12T21:51:48.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:47 smithi114 ceph-mon[25759]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "315f320f-95fd-4b18-ab6a-417ba7651667"}]': finished 2024-04-12T21:51:48.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:47 smithi114 ceph-mon[25759]: osdmap e5: 1 total, 0 up, 1 in 2024-04-12T21:51:48.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:47 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T21:51:49.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:48 smithi097 ceph-mon[24835]: mgrmap e14: y(active, since 64s), standbys: x 2024-04-12T21:51:49.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:48 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-12T21:51:49.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:48 smithi097 ceph-mon[24835]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:49.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:48 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/2431460262' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:51:49.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:48 smithi097 ceph-mon[29620]: mgrmap e14: y(active, since 64s), standbys: x 2024-04-12T21:51:49.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:48 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-12T21:51:49.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:48 smithi097 ceph-mon[29620]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:49.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:48 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/2431460262' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:51:49.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:48 smithi114 ceph-mon[25759]: mgrmap e14: y(active, since 64s), standbys: x 2024-04-12T21:51:49.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:48 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-12T21:51:49.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:48 smithi114 ceph-mon[25759]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:49.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:48 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/2431460262' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:51:51.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:51 smithi097 ceph-mon[24835]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:51.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:51 smithi097 ceph-mon[29620]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:51.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:51 smithi114 ceph-mon[25759]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:53.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:53 smithi097 ceph-mon[24835]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:53.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:53 smithi097 ceph-mon[29620]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:53.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:53 smithi114 ceph-mon[25759]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:54.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:54 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-12T21:51:54.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:54 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:51:54.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:54 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-12T21:51:54.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:54 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:51:54.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:54 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-12T21:51:54.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:54 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:51:55.344 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:55 smithi097 ceph-mon[24835]: Deploying daemon osd.0 on smithi097 2024-04-12T21:51:55.344 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:55 smithi097 ceph-mon[24835]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:55.344 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:55 smithi097 ceph-mon[29620]: Deploying daemon osd.0 on smithi097 2024-04-12T21:51:55.344 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:55 smithi097 ceph-mon[29620]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:55.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:55 smithi114 ceph-mon[25759]: Deploying daemon osd.0 on smithi097 2024-04-12T21:51:55.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:55 smithi114 ceph-mon[25759]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:57.226 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:57 smithi097 ceph-mon[29620]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:57.227 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:57 smithi097 ceph-mon[24835]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:57.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:57 smithi114 ceph-mon[25759]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:59.359 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:59 smithi097 ceph-mon[24835]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:59.359 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:59 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:59.359 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:59 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:51:59.359 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:59 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:51:59.359 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:51:59 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:51:59.359 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:59 smithi097 ceph-mon[29620]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:59.359 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:59 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:59.359 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:59 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:51:59.359 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:59 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:51:59.360 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:51:59 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:51:59.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:59 smithi114 ceph-mon[25759]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:51:59.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:59 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:51:59.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:59 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:51:59.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:59 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:51:59.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:51:59 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:52:01.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:01 smithi097 ceph-mon[29620]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:52:01.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:01 smithi097 ceph-mon[24835]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:52:01.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:01 smithi114 ceph-mon[25759]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:52:01.899 INFO:teuthology.orchestra.run.smithi097.stdout:Created osd(s) 0 on host 'smithi097' 2024-04-12T21:52:02.536 DEBUG:teuthology.orchestra.run.smithi097:osd.0> sudo journalctl -f -n 0 -u ceph-80bf7680-f916-11ee-bc8d-c7b262605968@osd.0.service 2024-04-12T21:52:02.538 INFO:tasks.cephadm:Deploying osd.1 on smithi097 with /dev/vg_nvme/lv_3... 2024-04-12T21:52:02.539 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-12T21:52:03.148 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:02 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:03.148 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:02 smithi097 ceph-mon[24835]: from='osd.0 [v2:172.21.15.97:6802/2413859098,v1:172.21.15.97:6803/2413859098]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-12T21:52:03.148 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:02 smithi097 ceph-mon[24835]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-12T21:52:03.149 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:02 smithi097 ceph-mon[24835]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:52:03.149 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:02 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:03.149 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:02 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:03.149 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:02 smithi097 ceph-mon[29620]: from='osd.0 [v2:172.21.15.97:6802/2413859098,v1:172.21.15.97:6803/2413859098]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-12T21:52:03.149 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:02 smithi097 ceph-mon[29620]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-12T21:52:03.149 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:02 smithi097 ceph-mon[29620]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:52:03.149 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:02 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:03.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:02 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:03.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:02 smithi114 ceph-mon[25759]: from='osd.0 [v2:172.21.15.97:6802/2413859098,v1:172.21.15.97:6803/2413859098]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-12T21:52:03.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:02 smithi114 ceph-mon[25759]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-12T21:52:03.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:02 smithi114 ceph-mon[25759]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:52:03.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:02 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:04.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:03 smithi097 ceph-mon[24835]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-12T21:52:04.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:03 smithi097 ceph-mon[24835]: osdmap e6: 1 total, 0 up, 1 in 2024-04-12T21:52:04.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:03 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T21:52:04.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:03 smithi097 ceph-mon[24835]: from='osd.0 [v2:172.21.15.97:6802/2413859098,v1:172.21.15.97:6803/2413859098]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-12T21:52:04.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:03 smithi097 ceph-mon[24835]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-12T21:52:04.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:03 smithi097 ceph-mon[29620]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-12T21:52:04.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:03 smithi097 ceph-mon[29620]: osdmap e6: 1 total, 0 up, 1 in 2024-04-12T21:52:04.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:03 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T21:52:04.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:03 smithi097 ceph-mon[29620]: from='osd.0 [v2:172.21.15.97:6802/2413859098,v1:172.21.15.97:6803/2413859098]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-12T21:52:04.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:03 smithi097 ceph-mon[29620]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-12T21:52:04.162 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 21:52:03 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-0[32474]: 2024-04-12T21:52:03.904+0000 7fe71572a700 -1 osd.0 0 waiting for initial osdmap 2024-04-12T21:52:04.162 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 21:52:03 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-0[32474]: 2024-04-12T21:52:03.907+0000 7fe70f0bf700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T21:52:04.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:03 smithi114 ceph-mon[25759]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-12T21:52:04.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:03 smithi114 ceph-mon[25759]: osdmap e6: 1 total, 0 up, 1 in 2024-04-12T21:52:04.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:03 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T21:52:04.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:03 smithi114 ceph-mon[25759]: from='osd.0 [v2:172.21.15.97:6802/2413859098,v1:172.21.15.97:6803/2413859098]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-12T21:52:04.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:03 smithi114 ceph-mon[25759]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-12T21:52:05.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:04 smithi097 ceph-mon[24835]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-04-12T21:52:05.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:04 smithi097 ceph-mon[24835]: osdmap e7: 1 total, 0 up, 1 in 2024-04-12T21:52:05.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:04 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T21:52:05.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:04 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T21:52:05.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:04 smithi097 ceph-mon[24835]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:52:05.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:04 smithi097 ceph-mon[29620]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-04-12T21:52:05.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:04 smithi097 ceph-mon[29620]: osdmap e7: 1 total, 0 up, 1 in 2024-04-12T21:52:05.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:04 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T21:52:05.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:04 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T21:52:05.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:04 smithi097 ceph-mon[29620]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:52:05.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:04 smithi114 ceph-mon[25759]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-04-12T21:52:05.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:04 smithi114 ceph-mon[25759]: osdmap e7: 1 total, 0 up, 1 in 2024-04-12T21:52:05.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:04 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T21:52:05.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:04 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T21:52:05.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:04 smithi114 ceph-mon[25759]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:52:05.910 INFO:teuthology.orchestra.run.smithi097.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-12T21:52:05.910 INFO:teuthology.orchestra.run.smithi097.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-12T21:52:05.910 INFO:teuthology.orchestra.run.smithi097.stderr: stderr: 10+0 records in 2024-04-12T21:52:05.910 INFO:teuthology.orchestra.run.smithi097.stderr:10+0 records out 2024-04-12T21:52:05.910 INFO:teuthology.orchestra.run.smithi097.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0123072 s, 852 MB/s 2024-04-12T21:52:05.911 INFO:teuthology.orchestra.run.smithi097.stderr:--> Zapping successful for: 2024-04-12T21:52:06.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:05 smithi097 ceph-mon[24835]: purged_snaps scrub starts 2024-04-12T21:52:06.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:05 smithi097 ceph-mon[24835]: purged_snaps scrub ok 2024-04-12T21:52:06.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:05 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T21:52:06.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:05 smithi097 ceph-mon[24835]: osd.0 [v2:172.21.15.97:6802/2413859098,v1:172.21.15.97:6803/2413859098] boot 2024-04-12T21:52:06.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:05 smithi097 ceph-mon[24835]: osdmap e8: 1 total, 1 up, 1 in 2024-04-12T21:52:06.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:05 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T21:52:06.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:05 smithi097 ceph-mon[29620]: purged_snaps scrub starts 2024-04-12T21:52:06.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:05 smithi097 ceph-mon[29620]: purged_snaps scrub ok 2024-04-12T21:52:06.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:05 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T21:52:06.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:05 smithi097 ceph-mon[29620]: osd.0 [v2:172.21.15.97:6802/2413859098,v1:172.21.15.97:6803/2413859098] boot 2024-04-12T21:52:06.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:05 smithi097 ceph-mon[29620]: osdmap e8: 1 total, 1 up, 1 in 2024-04-12T21:52:06.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:05 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T21:52:06.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:05 smithi114 ceph-mon[25759]: purged_snaps scrub starts 2024-04-12T21:52:06.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:05 smithi114 ceph-mon[25759]: purged_snaps scrub ok 2024-04-12T21:52:06.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:05 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T21:52:06.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:05 smithi114 ceph-mon[25759]: osd.0 [v2:172.21.15.97:6802/2413859098,v1:172.21.15.97:6803/2413859098] boot 2024-04-12T21:52:06.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:05 smithi114 ceph-mon[25759]: osdmap e8: 1 total, 1 up, 1 in 2024-04-12T21:52:06.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:05 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T21:52:06.584 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- ceph orch daemon add osd smithi097:vg_nvme/lv_3 2024-04-12T21:52:07.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:06 smithi097 ceph-mon[29620]: osdmap e9: 1 total, 1 up, 1 in 2024-04-12T21:52:07.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:06 smithi097 ceph-mon[29620]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:52:07.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:06 smithi097 ceph-mon[24835]: osdmap e9: 1 total, 1 up, 1 in 2024-04-12T21:52:07.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:06 smithi097 ceph-mon[24835]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:52:07.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:06 smithi114 ceph-mon[25759]: osdmap e9: 1 total, 1 up, 1 in 2024-04-12T21:52:07.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:06 smithi114 ceph-mon[25759]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:52:08.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:08 smithi114 ceph-mon[25759]: Detected new or changed devices on smithi097 2024-04-12T21:52:08.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:08 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:08.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:08 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:52:08.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:08 smithi114 ceph-mon[25759]: Adjusting osd_memory_target on smithi097 to 16064M 2024-04-12T21:52:08.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:08 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:08.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:08 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:08.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:08 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:52:08.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:08 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:52:08.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:08 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:52:08.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:08 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:08.569 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:08 smithi097 ceph-mon[24835]: Detected new or changed devices on smithi097 2024-04-12T21:52:08.569 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:08 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:08.569 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:08 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:52:08.569 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:08 smithi097 ceph-mon[24835]: Adjusting osd_memory_target on smithi097 to 16064M 2024-04-12T21:52:08.570 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:08 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:08.570 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:08 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:08.570 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:08 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:52:08.570 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:08 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:52:08.570 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:08 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:52:08.570 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:08 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:08.570 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:08 smithi097 ceph-mon[29620]: Detected new or changed devices on smithi097 2024-04-12T21:52:08.570 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:08 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:08.570 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:08 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:52:08.570 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:08 smithi097 ceph-mon[29620]: Adjusting osd_memory_target on smithi097 to 16064M 2024-04-12T21:52:08.570 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:08 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:08.571 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:08 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:08.571 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:08 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:52:08.571 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:08 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:52:08.571 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:08 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:52:08.571 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:08 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:09.362 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:09 smithi097 ceph-mon[24835]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:52:09.362 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:09 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:52:09.362 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:09 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:52:09.363 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:09 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:52:09.363 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:09 smithi097 ceph-mon[29620]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:52:09.363 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:09 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:52:09.363 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:09 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:52:09.363 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:09 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:52:09.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:09 smithi114 ceph-mon[25759]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:52:09.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:09 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:52:09.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:09 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:52:09.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:09 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:52:10.480 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:10 smithi097 ceph-mon[24835]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:52:10.481 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:10 smithi097 ceph-mon[29620]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:52:10.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:10 smithi114 ceph-mon[25759]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:52:11.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:11 smithi114 ceph-mon[25759]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:52:11.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:11 smithi097 ceph-mon[24835]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:52:11.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:11 smithi097 ceph-mon[29620]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:52:12.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:12 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/406410081' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c58dc94b-86d4-4b83-95ad-d3779a5aa46e"}]: dispatch 2024-04-12T21:52:12.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:12 smithi114 ceph-mon[25759]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c58dc94b-86d4-4b83-95ad-d3779a5aa46e"}]: dispatch 2024-04-12T21:52:12.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:12 smithi114 ceph-mon[25759]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c58dc94b-86d4-4b83-95ad-d3779a5aa46e"}]': finished 2024-04-12T21:52:12.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:12 smithi114 ceph-mon[25759]: osdmap e10: 2 total, 1 up, 2 in 2024-04-12T21:52:12.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:12 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T21:52:12.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:12 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/1083005774' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:52:12.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:12 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/406410081' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c58dc94b-86d4-4b83-95ad-d3779a5aa46e"}]: dispatch 2024-04-12T21:52:12.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:12 smithi097 ceph-mon[24835]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c58dc94b-86d4-4b83-95ad-d3779a5aa46e"}]: dispatch 2024-04-12T21:52:12.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:12 smithi097 ceph-mon[24835]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c58dc94b-86d4-4b83-95ad-d3779a5aa46e"}]': finished 2024-04-12T21:52:12.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:12 smithi097 ceph-mon[24835]: osdmap e10: 2 total, 1 up, 2 in 2024-04-12T21:52:12.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:12 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T21:52:12.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:12 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/1083005774' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:52:12.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:12 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/406410081' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c58dc94b-86d4-4b83-95ad-d3779a5aa46e"}]: dispatch 2024-04-12T21:52:12.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:12 smithi097 ceph-mon[29620]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c58dc94b-86d4-4b83-95ad-d3779a5aa46e"}]: dispatch 2024-04-12T21:52:12.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:12 smithi097 ceph-mon[29620]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c58dc94b-86d4-4b83-95ad-d3779a5aa46e"}]': finished 2024-04-12T21:52:12.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:12 smithi097 ceph-mon[29620]: osdmap e10: 2 total, 1 up, 2 in 2024-04-12T21:52:12.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:12 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T21:52:12.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:12 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/1083005774' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:52:13.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:13 smithi097 ceph-mon[29620]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:52:13.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:13 smithi097 ceph-mon[24835]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:52:13.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:13 smithi114 ceph-mon[25759]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:52:15.519 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:15 smithi097 ceph-mon[24835]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:52:15.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:15 smithi097 ceph-mon[29620]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:52:15.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:15 smithi114 ceph-mon[25759]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:52:17.595 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:17 smithi097 ceph-mon[24835]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:52:17.595 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:17 smithi097 ceph-mon[29620]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:52:17.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:17 smithi114 ceph-mon[25759]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:52:18.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:18 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-12T21:52:18.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:18 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:52:18.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:18 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-12T21:52:18.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:18 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:52:18.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:18 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-12T21:52:18.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:18 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:52:19.554 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:19 smithi097 ceph-mon[24835]: Deploying daemon osd.1 on smithi097 2024-04-12T21:52:19.555 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:19 smithi097 ceph-mon[24835]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:52:19.555 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:19 smithi097 ceph-mon[29620]: Deploying daemon osd.1 on smithi097 2024-04-12T21:52:19.555 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:19 smithi097 ceph-mon[29620]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:52:19.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:19 smithi114 ceph-mon[25759]: Deploying daemon osd.1 on smithi097 2024-04-12T21:52:19.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:19 smithi114 ceph-mon[25759]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:52:21.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:21 smithi097 ceph-mon[24835]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:52:21.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:21 smithi097 ceph-mon[29620]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:52:21.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:21 smithi114 ceph-mon[25759]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:52:23.489 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:23 smithi097 ceph-mon[24835]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:52:23.489 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:23 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:23.489 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:23 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:52:23.489 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:23 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:52:23.489 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:23 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:52:23.489 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:23 smithi097 ceph-mon[29620]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:52:23.490 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:23 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:23.490 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:23 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:52:23.490 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:23 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:52:23.490 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:23 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:52:23.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:23 smithi114 ceph-mon[25759]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:52:23.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:23 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:23.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:23 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:52:23.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:23 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:52:23.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:23 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:52:25.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:25 smithi097 ceph-mon[24835]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:52:25.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:25 smithi097 ceph-mon[29620]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:52:25.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:25 smithi114 ceph-mon[25759]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:52:26.040 INFO:teuthology.orchestra.run.smithi097.stdout:Created osd(s) 1 on host 'smithi097' 2024-04-12T21:52:26.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:26 smithi097 ceph-mon[24835]: from='osd.1 [v2:172.21.15.97:6810/1786985012,v1:172.21.15.97:6811/1786985012]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-12T21:52:26.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:26 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:26.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:26 smithi097 ceph-mon[29620]: from='osd.1 [v2:172.21.15.97:6810/1786985012,v1:172.21.15.97:6811/1786985012]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-12T21:52:26.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:26 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:26.723 DEBUG:teuthology.orchestra.run.smithi097:osd.1> sudo journalctl -f -n 0 -u ceph-80bf7680-f916-11ee-bc8d-c7b262605968@osd.1.service 2024-04-12T21:52:26.726 INFO:tasks.cephadm:Deploying osd.2 on smithi097 with /dev/vg_nvme/lv_2... 2024-04-12T21:52:26.726 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-12T21:52:26.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:26 smithi114 ceph-mon[25759]: from='osd.1 [v2:172.21.15.97:6810/1786985012,v1:172.21.15.97:6811/1786985012]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-12T21:52:26.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:26 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:27.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:27 smithi097 ceph-mon[24835]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:52:27.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:27 smithi097 ceph-mon[24835]: from='osd.1 [v2:172.21.15.97:6810/1786985012,v1:172.21.15.97:6811/1786985012]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-12T21:52:27.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:27 smithi097 ceph-mon[24835]: osdmap e11: 2 total, 1 up, 2 in 2024-04-12T21:52:27.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:27 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T21:52:27.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:27 smithi097 ceph-mon[24835]: from='osd.1 [v2:172.21.15.97:6810/1786985012,v1:172.21.15.97:6811/1786985012]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-12T21:52:27.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:27 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:27.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:27 smithi097 ceph-mon[29620]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:52:27.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:27 smithi097 ceph-mon[29620]: from='osd.1 [v2:172.21.15.97:6810/1786985012,v1:172.21.15.97:6811/1786985012]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-12T21:52:27.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:27 smithi097 ceph-mon[29620]: osdmap e11: 2 total, 1 up, 2 in 2024-04-12T21:52:27.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:27 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T21:52:27.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:27 smithi097 ceph-mon[29620]: from='osd.1 [v2:172.21.15.97:6810/1786985012,v1:172.21.15.97:6811/1786985012]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-12T21:52:27.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:27 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:27.662 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 21:52:27 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-1[35441]: 2024-04-12T21:52:27.357+0000 7fdc2e1fb700 -1 osd.1 0 waiting for initial osdmap 2024-04-12T21:52:27.662 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 21:52:27 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-1[35441]: 2024-04-12T21:52:27.360+0000 7fdc2ab96700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T21:52:27.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:27 smithi114 ceph-mon[25759]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:52:27.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:27 smithi114 ceph-mon[25759]: from='osd.1 [v2:172.21.15.97:6810/1786985012,v1:172.21.15.97:6811/1786985012]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-12T21:52:27.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:27 smithi114 ceph-mon[25759]: osdmap e11: 2 total, 1 up, 2 in 2024-04-12T21:52:27.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:27 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T21:52:27.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:27 smithi114 ceph-mon[25759]: from='osd.1 [v2:172.21.15.97:6810/1786985012,v1:172.21.15.97:6811/1786985012]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-12T21:52:27.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:27 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:28.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:28 smithi097 ceph-mon[29620]: from='osd.1 [v2:172.21.15.97:6810/1786985012,v1:172.21.15.97:6811/1786985012]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-04-12T21:52:28.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:28 smithi097 ceph-mon[29620]: osdmap e12: 2 total, 1 up, 2 in 2024-04-12T21:52:28.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:28 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T21:52:28.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:28 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T21:52:28.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:28 smithi097 ceph-mon[24835]: from='osd.1 [v2:172.21.15.97:6810/1786985012,v1:172.21.15.97:6811/1786985012]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-04-12T21:52:28.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:28 smithi097 ceph-mon[24835]: osdmap e12: 2 total, 1 up, 2 in 2024-04-12T21:52:28.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:28 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T21:52:28.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:28 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T21:52:28.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:28 smithi114 ceph-mon[25759]: from='osd.1 [v2:172.21.15.97:6810/1786985012,v1:172.21.15.97:6811/1786985012]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-04-12T21:52:28.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:28 smithi114 ceph-mon[25759]: osdmap e12: 2 total, 1 up, 2 in 2024-04-12T21:52:28.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:28 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T21:52:28.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:28 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T21:52:29.441 INFO:teuthology.orchestra.run.smithi097.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-12T21:52:29.442 INFO:teuthology.orchestra.run.smithi097.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-12T21:52:29.442 INFO:teuthology.orchestra.run.smithi097.stderr: stderr: 10+0 records in 2024-04-12T21:52:29.442 INFO:teuthology.orchestra.run.smithi097.stderr:10+0 records out 2024-04-12T21:52:29.442 INFO:teuthology.orchestra.run.smithi097.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0128538 s, 816 MB/s 2024-04-12T21:52:29.442 INFO:teuthology.orchestra.run.smithi097.stderr:--> Zapping successful for: 2024-04-12T21:52:29.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:29 smithi097 ceph-mon[29620]: purged_snaps scrub starts 2024-04-12T21:52:29.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:29 smithi097 ceph-mon[29620]: purged_snaps scrub ok 2024-04-12T21:52:29.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:29 smithi097 ceph-mon[29620]: pgmap v54: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:52:29.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:29 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T21:52:29.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:29 smithi097 ceph-mon[29620]: osd.1 [v2:172.21.15.97:6810/1786985012,v1:172.21.15.97:6811/1786985012] boot 2024-04-12T21:52:29.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:29 smithi097 ceph-mon[29620]: osdmap e13: 2 total, 2 up, 2 in 2024-04-12T21:52:29.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:29 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T21:52:29.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:29 smithi097 ceph-mon[24835]: purged_snaps scrub starts 2024-04-12T21:52:29.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:29 smithi097 ceph-mon[24835]: purged_snaps scrub ok 2024-04-12T21:52:29.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:29 smithi097 ceph-mon[24835]: pgmap v54: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:52:29.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:29 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T21:52:29.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:29 smithi097 ceph-mon[24835]: osd.1 [v2:172.21.15.97:6810/1786985012,v1:172.21.15.97:6811/1786985012] boot 2024-04-12T21:52:29.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:29 smithi097 ceph-mon[24835]: osdmap e13: 2 total, 2 up, 2 in 2024-04-12T21:52:29.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:29 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T21:52:29.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:29 smithi114 ceph-mon[25759]: purged_snaps scrub starts 2024-04-12T21:52:29.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:29 smithi114 ceph-mon[25759]: purged_snaps scrub ok 2024-04-12T21:52:29.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:29 smithi114 ceph-mon[25759]: pgmap v54: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:52:29.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:29 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T21:52:29.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:29 smithi114 ceph-mon[25759]: osd.1 [v2:172.21.15.97:6810/1786985012,v1:172.21.15.97:6811/1786985012] boot 2024-04-12T21:52:29.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:29 smithi114 ceph-mon[25759]: osdmap e13: 2 total, 2 up, 2 in 2024-04-12T21:52:29.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:29 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T21:52:30.201 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- ceph orch daemon add osd smithi097:vg_nvme/lv_2 2024-04-12T21:52:30.481 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:30 smithi097 ceph-mon[24835]: osdmap e14: 2 total, 2 up, 2 in 2024-04-12T21:52:30.481 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:30 smithi097 ceph-mon[29620]: osdmap e14: 2 total, 2 up, 2 in 2024-04-12T21:52:30.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:30 smithi114 ceph-mon[25759]: osdmap e14: 2 total, 2 up, 2 in 2024-04-12T21:52:31.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:31 smithi097 ceph-mon[24835]: pgmap v57: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:52:31.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:31 smithi097 ceph-mon[29620]: pgmap v57: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:52:31.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:31 smithi114 ceph-mon[25759]: pgmap v57: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:52:33.564 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:33 smithi097 ceph-mon[24835]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:52:33.565 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:33 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:33.565 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:33 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:52:33.565 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:33 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:52:33.565 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:33 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:33.565 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:33 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:33.565 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:33 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:52:33.565 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:33 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:52:33.565 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:33 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:52:33.565 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:33 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:33.565 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:33 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:52:33.565 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:33 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:52:33.565 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:33 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:52:33.566 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:33 smithi097 ceph-mon[29620]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:52:33.566 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:33 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:33.566 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:33 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:52:33.566 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:33 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:52:33.566 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:33 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:33.566 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:33 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:33.566 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:33 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:52:33.566 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:33 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:52:33.566 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:33 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:52:33.566 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:33 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:33.566 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:33 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:52:33.566 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:33 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:52:33.567 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:33 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:52:33.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:33 smithi114 ceph-mon[25759]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:52:33.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:33 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:33.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:33 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:52:33.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:33 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:52:33.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:33 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:33.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:33 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:33.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:33 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:52:33.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:33 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:52:33.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:33 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:52:33.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:33 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:33.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:33 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:52:33.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:33 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:52:33.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:33 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:52:34.499 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:34 smithi097 ceph-mon[24835]: Detected new or changed devices on smithi097 2024-04-12T21:52:34.499 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:34 smithi097 ceph-mon[24835]: Adjusting osd_memory_target on smithi097 to 8032M 2024-04-12T21:52:34.499 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:34 smithi097 ceph-mon[24835]: from='client.24178 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:52:34.499 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:34 smithi097 ceph-mon[29620]: Detected new or changed devices on smithi097 2024-04-12T21:52:34.499 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:34 smithi097 ceph-mon[29620]: Adjusting osd_memory_target on smithi097 to 8032M 2024-04-12T21:52:34.499 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:34 smithi097 ceph-mon[29620]: from='client.24178 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:52:34.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:34 smithi114 ceph-mon[25759]: Detected new or changed devices on smithi097 2024-04-12T21:52:34.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:34 smithi114 ceph-mon[25759]: Adjusting osd_memory_target on smithi097 to 8032M 2024-04-12T21:52:34.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:34 smithi114 ceph-mon[25759]: from='client.24178 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:52:35.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:35 smithi097 ceph-mon[24835]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:52:35.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:35 smithi097 ceph-mon[29620]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:52:35.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:35 smithi114 ceph-mon[25759]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:52:36.569 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:36 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/3207926412' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4920ee2a-de4d-4072-92bb-dae616a76152"}]: dispatch 2024-04-12T21:52:36.569 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:36 smithi097 ceph-mon[29620]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4920ee2a-de4d-4072-92bb-dae616a76152"}]: dispatch 2024-04-12T21:52:36.569 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:36 smithi097 ceph-mon[29620]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4920ee2a-de4d-4072-92bb-dae616a76152"}]': finished 2024-04-12T21:52:36.569 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:36 smithi097 ceph-mon[29620]: osdmap e15: 3 total, 2 up, 3 in 2024-04-12T21:52:36.569 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:36 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T21:52:36.569 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:36 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/2575060008' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:52:36.569 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:36 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/3207926412' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4920ee2a-de4d-4072-92bb-dae616a76152"}]: dispatch 2024-04-12T21:52:36.569 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:36 smithi097 ceph-mon[24835]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4920ee2a-de4d-4072-92bb-dae616a76152"}]: dispatch 2024-04-12T21:52:36.569 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:36 smithi097 ceph-mon[24835]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4920ee2a-de4d-4072-92bb-dae616a76152"}]': finished 2024-04-12T21:52:36.569 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:36 smithi097 ceph-mon[24835]: osdmap e15: 3 total, 2 up, 3 in 2024-04-12T21:52:36.570 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:36 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T21:52:36.570 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:36 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/2575060008' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:52:36.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:36 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/3207926412' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4920ee2a-de4d-4072-92bb-dae616a76152"}]: dispatch 2024-04-12T21:52:36.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:36 smithi114 ceph-mon[25759]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4920ee2a-de4d-4072-92bb-dae616a76152"}]: dispatch 2024-04-12T21:52:36.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:36 smithi114 ceph-mon[25759]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4920ee2a-de4d-4072-92bb-dae616a76152"}]': finished 2024-04-12T21:52:36.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:36 smithi114 ceph-mon[25759]: osdmap e15: 3 total, 2 up, 3 in 2024-04-12T21:52:36.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:36 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T21:52:36.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:36 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/2575060008' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:52:37.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:37 smithi097 ceph-mon[24835]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:52:37.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:37 smithi097 ceph-mon[29620]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:52:37.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:37 smithi114 ceph-mon[25759]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:52:39.353 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:39 smithi097 ceph-mon[24835]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:52:39.353 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:39 smithi097 ceph-mon[29620]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:52:39.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:39 smithi114 ceph-mon[25759]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:52:41.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:41 smithi097 ceph-mon[24835]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:52:41.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:41 smithi097 ceph-mon[29620]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:52:41.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:41 smithi114 ceph-mon[25759]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:52:42.372 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:42 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-12T21:52:42.372 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:42 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:52:42.372 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:42 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-12T21:52:42.372 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:42 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:52:42.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:42 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-12T21:52:42.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:42 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:52:43.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:43 smithi097 ceph-mon[24835]: Deploying daemon osd.2 on smithi097 2024-04-12T21:52:43.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:43 smithi097 ceph-mon[24835]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:52:43.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:43 smithi097 ceph-mon[29620]: Deploying daemon osd.2 on smithi097 2024-04-12T21:52:43.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:43 smithi097 ceph-mon[29620]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:52:43.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:43 smithi114 ceph-mon[25759]: Deploying daemon osd.2 on smithi097 2024-04-12T21:52:43.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:43 smithi114 ceph-mon[25759]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:52:45.291 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:45 smithi097 ceph-mon[24835]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:52:45.291 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:45 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T21:52:45.291 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:45 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T21:52:45.292 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:45 smithi097 ceph-mon[29620]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:52:45.292 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:45 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T21:52:45.292 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:45 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T21:52:45.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:45 smithi114 ceph-mon[25759]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:52:45.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:45 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T21:52:45.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:45 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T21:52:47.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:47 smithi097 ceph-mon[24835]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:52:47.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:47 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:47.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:47 smithi097 ceph-mon[29620]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:52:47.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:47 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:47.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:47 smithi114 ceph-mon[25759]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:52:47.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:47 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:48.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:48 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:52:48.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:48 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:52:48.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:48 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:52:48.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:48 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:52:48.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:48 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:52:48.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:48 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:52:48.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:48 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:52:48.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:48 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:52:48.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:48 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:52:49.369 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:49 smithi097 ceph-mon[24835]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:52:49.369 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:49 smithi097 ceph-mon[29620]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:52:49.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:49 smithi114 ceph-mon[25759]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:52:50.390 INFO:teuthology.orchestra.run.smithi097.stdout:Created osd(s) 2 on host 'smithi097' 2024-04-12T21:52:51.046 DEBUG:teuthology.orchestra.run.smithi097:osd.2> sudo journalctl -f -n 0 -u ceph-80bf7680-f916-11ee-bc8d-c7b262605968@osd.2.service 2024-04-12T21:52:51.048 INFO:tasks.cephadm:Deploying osd.3 on smithi097 with /dev/vg_nvme/lv_1... 2024-04-12T21:52:51.048 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-12T21:52:51.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:51 smithi097 ceph-mon[24835]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:52:51.409 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:51 smithi097 ceph-mon[24835]: from='osd.2 [v2:172.21.15.97:6818/1944472297,v1:172.21.15.97:6819/1944472297]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-12T21:52:51.409 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:51 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:51.409 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:51 smithi097 ceph-mon[29620]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:52:51.409 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:51 smithi097 ceph-mon[29620]: from='osd.2 [v2:172.21.15.97:6818/1944472297,v1:172.21.15.97:6819/1944472297]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-12T21:52:51.409 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:51 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:51.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:51 smithi114 ceph-mon[25759]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:52:51.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:51 smithi114 ceph-mon[25759]: from='osd.2 [v2:172.21.15.97:6818/1944472297,v1:172.21.15.97:6819/1944472297]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-12T21:52:51.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:51 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:52.363 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:52 smithi097 ceph-mon[24835]: from='osd.2 [v2:172.21.15.97:6818/1944472297,v1:172.21.15.97:6819/1944472297]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-12T21:52:52.363 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:52 smithi097 ceph-mon[24835]: osdmap e16: 3 total, 2 up, 3 in 2024-04-12T21:52:52.363 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:52 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T21:52:52.363 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:52 smithi097 ceph-mon[24835]: from='osd.2 [v2:172.21.15.97:6818/1944472297,v1:172.21.15.97:6819/1944472297]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-12T21:52:52.363 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:52 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:52.364 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:52 smithi097 ceph-mon[29620]: from='osd.2 [v2:172.21.15.97:6818/1944472297,v1:172.21.15.97:6819/1944472297]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-12T21:52:52.364 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:52 smithi097 ceph-mon[29620]: osdmap e16: 3 total, 2 up, 3 in 2024-04-12T21:52:52.364 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:52 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T21:52:52.364 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:52 smithi097 ceph-mon[29620]: from='osd.2 [v2:172.21.15.97:6818/1944472297,v1:172.21.15.97:6819/1944472297]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-12T21:52:52.364 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:52 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:52.364 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 21:52:52 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-2[38344]: 2024-04-12T21:52:52.118+0000 7f83ff574700 -1 osd.2 0 waiting for initial osdmap 2024-04-12T21:52:52.364 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 21:52:52 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-2[38344]: 2024-04-12T21:52:52.122+0000 7f83fbf0f700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T21:52:52.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:52 smithi114 ceph-mon[25759]: from='osd.2 [v2:172.21.15.97:6818/1944472297,v1:172.21.15.97:6819/1944472297]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-12T21:52:52.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:52 smithi114 ceph-mon[25759]: osdmap e16: 3 total, 2 up, 3 in 2024-04-12T21:52:52.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:52 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T21:52:52.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:52 smithi114 ceph-mon[25759]: from='osd.2 [v2:172.21.15.97:6818/1944472297,v1:172.21.15.97:6819/1944472297]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-12T21:52:52.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:52 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:53.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:53 smithi097 ceph-mon[24835]: pgmap v70: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:52:53.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:53 smithi097 ceph-mon[24835]: from='osd.2 [v2:172.21.15.97:6818/1944472297,v1:172.21.15.97:6819/1944472297]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-04-12T21:52:53.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:53 smithi097 ceph-mon[24835]: osdmap e17: 3 total, 2 up, 3 in 2024-04-12T21:52:53.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:53 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T21:52:53.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:53 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T21:52:53.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:53 smithi097 ceph-mon[29620]: pgmap v70: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:52:53.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:53 smithi097 ceph-mon[29620]: from='osd.2 [v2:172.21.15.97:6818/1944472297,v1:172.21.15.97:6819/1944472297]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-04-12T21:52:53.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:53 smithi097 ceph-mon[29620]: osdmap e17: 3 total, 2 up, 3 in 2024-04-12T21:52:53.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:53 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T21:52:53.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:53 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T21:52:53.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:53 smithi114 ceph-mon[25759]: pgmap v70: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:52:53.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:53 smithi114 ceph-mon[25759]: from='osd.2 [v2:172.21.15.97:6818/1944472297,v1:172.21.15.97:6819/1944472297]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-04-12T21:52:53.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:53 smithi114 ceph-mon[25759]: osdmap e17: 3 total, 2 up, 3 in 2024-04-12T21:52:53.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:53 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T21:52:53.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:53 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T21:52:54.232 INFO:teuthology.orchestra.run.smithi097.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-12T21:52:54.232 INFO:teuthology.orchestra.run.smithi097.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-12T21:52:54.232 INFO:teuthology.orchestra.run.smithi097.stderr: stderr: 10+0 records in 2024-04-12T21:52:54.232 INFO:teuthology.orchestra.run.smithi097.stderr:10+0 records out 2024-04-12T21:52:54.232 INFO:teuthology.orchestra.run.smithi097.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.014957 s, 701 MB/s 2024-04-12T21:52:54.232 INFO:teuthology.orchestra.run.smithi097.stderr:--> Zapping successful for: 2024-04-12T21:52:54.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:54 smithi097 ceph-mon[24835]: purged_snaps scrub starts 2024-04-12T21:52:54.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:54 smithi097 ceph-mon[24835]: purged_snaps scrub ok 2024-04-12T21:52:54.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:54 smithi097 ceph-mon[24835]: osd.2 [v2:172.21.15.97:6818/1944472297,v1:172.21.15.97:6819/1944472297] boot 2024-04-12T21:52:54.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:54 smithi097 ceph-mon[24835]: osdmap e18: 3 total, 3 up, 3 in 2024-04-12T21:52:54.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:54 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T21:52:54.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:54 smithi097 ceph-mon[29620]: purged_snaps scrub starts 2024-04-12T21:52:54.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:54 smithi097 ceph-mon[29620]: purged_snaps scrub ok 2024-04-12T21:52:54.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:54 smithi097 ceph-mon[29620]: osd.2 [v2:172.21.15.97:6818/1944472297,v1:172.21.15.97:6819/1944472297] boot 2024-04-12T21:52:54.413 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:54 smithi097 ceph-mon[29620]: osdmap e18: 3 total, 3 up, 3 in 2024-04-12T21:52:54.413 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:54 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T21:52:54.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:54 smithi114 ceph-mon[25759]: purged_snaps scrub starts 2024-04-12T21:52:54.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:54 smithi114 ceph-mon[25759]: purged_snaps scrub ok 2024-04-12T21:52:54.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:54 smithi114 ceph-mon[25759]: osd.2 [v2:172.21.15.97:6818/1944472297,v1:172.21.15.97:6819/1944472297] boot 2024-04-12T21:52:54.495 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:54 smithi114 ceph-mon[25759]: osdmap e18: 3 total, 3 up, 3 in 2024-04-12T21:52:54.495 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:54 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T21:52:54.921 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- ceph orch daemon add osd smithi097:vg_nvme/lv_1 2024-04-12T21:52:55.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:55 smithi097 ceph-mon[24835]: pgmap v73: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:52:55.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:55 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-12T21:52:55.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:55 smithi097 ceph-mon[29620]: pgmap v73: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:52:55.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:55 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-12T21:52:55.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:55 smithi114 ceph-mon[25759]: pgmap v73: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:52:55.395 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:55 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-12T21:52:56.338 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:56 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-12T21:52:56.338 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:56 smithi097 ceph-mon[24835]: osdmap e19: 3 total, 3 up, 3 in 2024-04-12T21:52:56.338 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:56 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-12T21:52:56.338 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:56 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-12T21:52:56.338 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:56 smithi097 ceph-mon[29620]: osdmap e19: 3 total, 3 up, 3 in 2024-04-12T21:52:56.338 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:56 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-12T21:52:56.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:56 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-12T21:52:56.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:56 smithi114 ceph-mon[25759]: osdmap e19: 3 total, 3 up, 3 in 2024-04-12T21:52:56.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:56 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-12T21:52:56.661 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 21:52:56 smithi097 sudo[39640]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-12T21:52:56.661 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 21:52:56 smithi097 sudo[39640]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-12T21:52:56.661 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 21:52:56 smithi097 sudo[39640]: pam_unix(sudo:session): session closed for user root 2024-04-12T21:52:57.046 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 21:52:56 smithi097 sudo[39643]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-12T21:52:57.046 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 21:52:56 smithi097 sudo[39643]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-12T21:52:57.046 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 21:52:56 smithi097 sudo[39643]: pam_unix(sudo:session): session closed for user root 2024-04-12T21:52:57.311 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 21:52:57 smithi097 sudo[39667]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-12T21:52:57.311 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 21:52:57 smithi097 sudo[39667]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-12T21:52:57.311 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 21:52:57 smithi097 sudo[39667]: pam_unix(sudo:session): session closed for user root 2024-04-12T21:52:57.312 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:57 smithi097 ceph-mon[24835]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:52:57.312 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:57 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-12T21:52:57.312 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:57 smithi097 ceph-mon[24835]: osdmap e20: 3 total, 3 up, 3 in 2024-04-12T21:52:57.312 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:57 smithi097 ceph-mon[24835]: Detected new or changed devices on smithi097 2024-04-12T21:52:57.312 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:57 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:57.312 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:57 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:52:57.312 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:57 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:52:57.313 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:57 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:52:57.313 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:57 smithi097 ceph-mon[24835]: Adjusting osd_memory_target on smithi097 to 5354M 2024-04-12T21:52:57.313 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:57 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:57.313 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:57 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:57.313 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:57 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:52:57.313 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:57 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:52:57.313 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:57 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:52:57.313 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:57 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:57.314 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:57 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:57.314 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:57 smithi097 ceph-mon[29620]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:52:57.314 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:57 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-12T21:52:57.314 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:57 smithi097 ceph-mon[29620]: osdmap e20: 3 total, 3 up, 3 in 2024-04-12T21:52:57.314 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:57 smithi097 ceph-mon[29620]: Detected new or changed devices on smithi097 2024-04-12T21:52:57.314 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:57 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:57.314 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:57 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:52:57.314 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:57 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:52:57.314 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:57 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:52:57.314 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:57 smithi097 ceph-mon[29620]: Adjusting osd_memory_target on smithi097 to 5354M 2024-04-12T21:52:57.315 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:57 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:57.315 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:57 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:57.315 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:57 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:52:57.315 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:57 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:52:57.315 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:57 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:52:57.315 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:57 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:57.315 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:57 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:57.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:57 smithi114 ceph-mon[25759]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:52:57.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:57 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-12T21:52:57.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:57 smithi114 ceph-mon[25759]: osdmap e20: 3 total, 3 up, 3 in 2024-04-12T21:52:57.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:57 smithi114 ceph-mon[25759]: Detected new or changed devices on smithi097 2024-04-12T21:52:57.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:57 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:57.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:57 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:52:57.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:57 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:52:57.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:57 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:52:57.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:57 smithi114 ceph-mon[25759]: Adjusting osd_memory_target on smithi097 to 5354M 2024-04-12T21:52:57.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:57 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:57.495 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:57 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:57.495 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:57 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:52:57.495 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:57 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:52:57.495 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:57 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:52:57.495 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:57 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:57.495 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:57 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:52:57.565 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 21:52:57 smithi097 sudo[39695]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-12T21:52:57.566 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 21:52:57 smithi097 sudo[39695]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-12T21:52:57.566 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 21:52:57 smithi097 sudo[39695]: pam_unix(sudo:session): session closed for user root 2024-04-12T21:52:57.829 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 21:52:57 smithi097 sudo[39735]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-12T21:52:57.829 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 21:52:57 smithi097 sudo[39735]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-12T21:52:57.829 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 21:52:57 smithi097 sudo[39735]: pam_unix(sudo:session): session closed for user root 2024-04-12T21:52:58.084 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:58 smithi097 sudo[39789]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-12T21:52:58.084 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:58 smithi097 sudo[39789]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-12T21:52:58.085 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 21:52:57 smithi097 sudo[39781]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-12T21:52:58.085 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 21:52:57 smithi097 sudo[39781]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-12T21:52:58.085 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 21:52:57 smithi097 sudo[39781]: pam_unix(sudo:session): session closed for user root 2024-04-12T21:52:58.334 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:58 smithi097 sudo[39789]: pam_unix(sudo:session): session closed for user root 2024-04-12T21:52:58.334 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:58 smithi097 sudo[39806]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-12T21:52:58.334 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:58 smithi097 sudo[39806]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-12T21:52:58.335 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:58 smithi097 sudo[39806]: pam_unix(sudo:session): session closed for user root 2024-04-12T21:52:58.587 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:58 smithi097 ceph-mon[24835]: osdmap e21: 3 total, 3 up, 3 in 2024-04-12T21:52:58.587 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:58 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:52:58.587 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:58 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:52:58.587 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:58 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:52:58.587 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:58 smithi097 ceph-mon[24835]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-12T21:52:58.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:58 smithi114 ceph-mon[25759]: osdmap e21: 3 total, 3 up, 3 in 2024-04-12T21:52:58.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:58 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:52:58.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:58 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:52:58.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:58 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:52:58.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:58 smithi114 ceph-mon[25759]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-12T21:52:58.845 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:58 smithi097 sudo[39810]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-12T21:52:58.846 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:58 smithi097 sudo[39810]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-12T21:52:58.846 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:58 smithi097 sudo[39810]: pam_unix(sudo:session): session closed for user root 2024-04-12T21:52:59.143 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:58 smithi097 sudo[39834]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-12T21:52:59.143 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:58 smithi097 sudo[39834]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-12T21:52:59.143 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:58 smithi097 sudo[39834]: pam_unix(sudo:session): session closed for user root 2024-04-12T21:52:59.143 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:58 smithi097 ceph-mon[29620]: osdmap e21: 3 total, 3 up, 3 in 2024-04-12T21:52:59.143 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:58 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:52:59.143 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:58 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:52:59.143 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:58 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:52:59.143 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:58 smithi097 ceph-mon[29620]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-12T21:52:59.343 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:59 smithi114 sudo[27262]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-12T21:52:59.343 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:59 smithi114 sudo[27262]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-12T21:52:59.343 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:59 smithi114 sudo[27262]: pam_unix(sudo:session): session closed for user root 2024-04-12T21:52:59.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:59 smithi097 ceph-mon[24835]: from='client.14310 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:52:59.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:59 smithi097 ceph-mon[24835]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:52:59.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:59 smithi097 ceph-mon[24835]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-12T21:52:59.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:59 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-12T21:52:59.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:59 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:52:59.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:59 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T21:52:59.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:59 smithi097 ceph-mon[24835]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-12T21:52:59.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:59 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-12T21:52:59.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:59 smithi097 ceph-mon[24835]: mgrmap e15: y(active, since 2m), standbys: x 2024-04-12T21:52:59.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:59 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:52:59.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:59 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T21:52:59.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:59 smithi097 ceph-mon[24835]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-12T21:52:59.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:59 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-12T21:52:59.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:59 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:52:59.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:52:59 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T21:52:59.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:59 smithi097 ceph-mon[29620]: from='client.14310 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:52:59.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:59 smithi097 ceph-mon[29620]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:52:59.413 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:59 smithi097 ceph-mon[29620]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-12T21:52:59.413 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:59 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-12T21:52:59.413 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:59 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:52:59.413 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:59 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T21:52:59.413 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:59 smithi097 ceph-mon[29620]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-12T21:52:59.413 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:59 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-12T21:52:59.413 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:59 smithi097 ceph-mon[29620]: mgrmap e15: y(active, since 2m), standbys: x 2024-04-12T21:52:59.413 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:59 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:52:59.413 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:59 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T21:52:59.413 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:59 smithi097 ceph-mon[29620]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-12T21:52:59.413 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:59 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-12T21:52:59.413 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:59 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:52:59.413 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:52:59 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T21:52:59.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:59 smithi114 sudo[27265]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-12T21:52:59.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:59 smithi114 sudo[27265]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-12T21:52:59.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:59 smithi114 sudo[27265]: pam_unix(sudo:session): session closed for user root 2024-04-12T21:52:59.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:59 smithi114 ceph-mon[25759]: from='client.14310 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:52:59.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:59 smithi114 ceph-mon[25759]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:52:59.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:59 smithi114 ceph-mon[25759]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-12T21:52:59.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:59 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-12T21:52:59.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:59 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:52:59.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:59 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T21:52:59.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:59 smithi114 ceph-mon[25759]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-12T21:52:59.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:59 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-12T21:52:59.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:59 smithi114 ceph-mon[25759]: mgrmap e15: y(active, since 2m), standbys: x 2024-04-12T21:52:59.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:59 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:52:59.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:59 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T21:52:59.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:59 smithi114 ceph-mon[25759]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-12T21:52:59.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:59 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-12T21:52:59.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:59 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:52:59.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:52:59 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T21:53:00.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:00 smithi097 ceph-mon[24835]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-12T21:53:00.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:00 smithi097 ceph-mon[24835]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-12T21:53:00.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:00 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:53:00.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:00 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/980456822' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "928b3bd7-c0df-443c-9a28-5ceb62ae3160"}]: dispatch 2024-04-12T21:53:00.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:00 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/980456822' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "928b3bd7-c0df-443c-9a28-5ceb62ae3160"}]': finished 2024-04-12T21:53:00.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:00 smithi097 ceph-mon[24835]: osdmap e22: 4 total, 3 up, 4 in 2024-04-12T21:53:00.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:00 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T21:53:00.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:00 smithi097 ceph-mon[29620]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-12T21:53:00.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:00 smithi097 ceph-mon[29620]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-12T21:53:00.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:00 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:53:00.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:00 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/980456822' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "928b3bd7-c0df-443c-9a28-5ceb62ae3160"}]: dispatch 2024-04-12T21:53:00.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:00 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/980456822' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "928b3bd7-c0df-443c-9a28-5ceb62ae3160"}]': finished 2024-04-12T21:53:00.663 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:00 smithi097 ceph-mon[29620]: osdmap e22: 4 total, 3 up, 4 in 2024-04-12T21:53:00.663 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:00 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T21:53:00.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:00 smithi114 ceph-mon[25759]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-12T21:53:00.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:00 smithi114 ceph-mon[25759]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-12T21:53:00.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:00 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:53:00.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:00 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/980456822' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "928b3bd7-c0df-443c-9a28-5ceb62ae3160"}]: dispatch 2024-04-12T21:53:00.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:00 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/980456822' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "928b3bd7-c0df-443c-9a28-5ceb62ae3160"}]': finished 2024-04-12T21:53:00.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:00 smithi114 ceph-mon[25759]: osdmap e22: 4 total, 3 up, 4 in 2024-04-12T21:53:00.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:00 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T21:53:01.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:01 smithi097 ceph-mon[24835]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:53:01.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:01 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/2633443819' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:53:01.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:01 smithi097 ceph-mon[29620]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:53:01.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:01 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/2633443819' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:53:01.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:01 smithi114 ceph-mon[25759]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:53:01.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:01 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/2633443819' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:53:03.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:03 smithi097 ceph-mon[24835]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:53:03.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:03 smithi097 ceph-mon[29620]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:53:03.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:03 smithi114 ceph-mon[25759]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:53:05.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:05 smithi097 ceph-mon[24835]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:53:05.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:05 smithi097 ceph-mon[29620]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:53:05.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:05 smithi114 ceph-mon[25759]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:53:07.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:07 smithi097 ceph-mon[24835]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 53 KiB/s wr, 4 op/s 2024-04-12T21:53:07.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:07 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-12T21:53:07.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:07 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:53:07.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:07 smithi097 ceph-mon[24835]: Deploying daemon osd.3 on smithi097 2024-04-12T21:53:07.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:07 smithi097 ceph-mon[29620]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 53 KiB/s wr, 4 op/s 2024-04-12T21:53:07.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:07 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-12T21:53:07.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:07 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:53:07.663 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:07 smithi097 ceph-mon[29620]: Deploying daemon osd.3 on smithi097 2024-04-12T21:53:07.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:07 smithi114 ceph-mon[25759]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 53 KiB/s wr, 4 op/s 2024-04-12T21:53:07.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:07 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-12T21:53:07.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:07 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:53:07.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:07 smithi114 ceph-mon[25759]: Deploying daemon osd.3 on smithi097 2024-04-12T21:53:09.646 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:09 smithi097 ceph-mon[24835]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 1.3 KiB/s rd, 40 KiB/s wr, 3 op/s 2024-04-12T21:53:09.647 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:09 smithi097 ceph-mon[29620]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 1.3 KiB/s rd, 40 KiB/s wr, 3 op/s 2024-04-12T21:53:09.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:09 smithi114 ceph-mon[25759]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 1.3 KiB/s rd, 40 KiB/s wr, 3 op/s 2024-04-12T21:53:11.637 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:11 smithi097 ceph-mon[29620]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 32 KiB/s wr, 2 op/s 2024-04-12T21:53:11.637 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:11 smithi097 ceph-mon[24835]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 32 KiB/s wr, 2 op/s 2024-04-12T21:53:11.673 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:11 smithi114 ceph-mon[25759]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 32 KiB/s wr, 2 op/s 2024-04-12T21:53:12.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:12 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:53:12.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:12 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:53:12.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:12 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:53:12.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:12 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:53:12.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:12 smithi097 ceph-mon[24835]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 938 B/s rd, 27 KiB/s wr, 2 op/s 2024-04-12T21:53:12.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:12 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:53:12.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:12 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:53:12.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:12 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:53:12.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:12 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:53:12.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:12 smithi097 ceph-mon[29620]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 938 B/s rd, 27 KiB/s wr, 2 op/s 2024-04-12T21:53:12.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:12 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:53:12.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:12 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:53:12.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:12 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:53:12.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:12 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:53:12.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:12 smithi114 ceph-mon[25759]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 938 B/s rd, 27 KiB/s wr, 2 op/s 2024-04-12T21:53:14.537 INFO:teuthology.orchestra.run.smithi097.stdout:Created osd(s) 3 on host 'smithi097' 2024-04-12T21:53:15.201 DEBUG:teuthology.orchestra.run.smithi097:osd.3> sudo journalctl -f -n 0 -u ceph-80bf7680-f916-11ee-bc8d-c7b262605968@osd.3.service 2024-04-12T21:53:15.204 INFO:tasks.cephadm:Deploying osd.4 on smithi114 with /dev/vg_nvme/lv_4... 2024-04-12T21:53:15.204 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-12T21:53:15.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:15 smithi114 ceph-mon[25759]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 938 B/s rd, 27 KiB/s wr, 2 op/s 2024-04-12T21:53:15.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:15 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:53:15.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:15 smithi114 ceph-mon[25759]: from='osd.3 [v2:172.21.15.97:6826/75554646,v1:172.21.15.97:6827/75554646]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-12T21:53:15.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:15 smithi114 ceph-mon[25759]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-12T21:53:15.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:15 smithi097 ceph-mon[24835]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 938 B/s rd, 27 KiB/s wr, 2 op/s 2024-04-12T21:53:15.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:15 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:53:15.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:15 smithi097 ceph-mon[24835]: from='osd.3 [v2:172.21.15.97:6826/75554646,v1:172.21.15.97:6827/75554646]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-12T21:53:15.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:15 smithi097 ceph-mon[24835]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-12T21:53:15.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:15 smithi097 ceph-mon[29620]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 938 B/s rd, 27 KiB/s wr, 2 op/s 2024-04-12T21:53:15.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:15 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:53:15.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:15 smithi097 ceph-mon[29620]: from='osd.3 [v2:172.21.15.97:6826/75554646,v1:172.21.15.97:6827/75554646]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-12T21:53:15.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:15 smithi097 ceph-mon[29620]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-12T21:53:16.707 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:16 smithi114 ceph-mon[25759]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-12T21:53:16.707 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:16 smithi114 ceph-mon[25759]: osdmap e23: 4 total, 3 up, 4 in 2024-04-12T21:53:16.707 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:16 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T21:53:16.707 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:16 smithi114 ceph-mon[25759]: from='osd.3 [v2:172.21.15.97:6826/75554646,v1:172.21.15.97:6827/75554646]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-12T21:53:16.707 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:16 smithi114 ceph-mon[25759]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-12T21:53:16.707 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:16 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:53:16.707 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:16 smithi114 ceph-mon[25759]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:53:16.822 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:16 smithi097 ceph-mon[29620]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-12T21:53:16.823 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:16 smithi097 ceph-mon[29620]: osdmap e23: 4 total, 3 up, 4 in 2024-04-12T21:53:16.823 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:16 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T21:53:16.823 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:16 smithi097 ceph-mon[29620]: from='osd.3 [v2:172.21.15.97:6826/75554646,v1:172.21.15.97:6827/75554646]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-12T21:53:16.823 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:16 smithi097 ceph-mon[29620]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-12T21:53:16.823 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:16 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:53:16.823 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:16 smithi097 ceph-mon[29620]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:53:16.823 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:16 smithi097 ceph-mon[24835]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-12T21:53:16.823 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:16 smithi097 ceph-mon[24835]: osdmap e23: 4 total, 3 up, 4 in 2024-04-12T21:53:16.824 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:16 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T21:53:16.824 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:16 smithi097 ceph-mon[24835]: from='osd.3 [v2:172.21.15.97:6826/75554646,v1:172.21.15.97:6827/75554646]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-12T21:53:16.824 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:16 smithi097 ceph-mon[24835]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-12T21:53:16.824 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:16 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:53:16.824 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:16 smithi097 ceph-mon[24835]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:53:16.824 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 21:53:16 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-3[41341]: 2024-04-12T21:53:16.542+0000 7f234bf74700 -1 osd.3 0 waiting for initial osdmap 2024-04-12T21:53:16.824 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 21:53:16 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-3[41341]: 2024-04-12T21:53:16.546+0000 7f234690b700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T21:53:17.073 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-12T21:53:17.073 INFO:teuthology.orchestra.run.smithi114.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-12T21:53:17.074 INFO:teuthology.orchestra.run.smithi114.stderr: stderr: 10+0 records in 2024-04-12T21:53:17.074 INFO:teuthology.orchestra.run.smithi114.stderr:10+0 records out 2024-04-12T21:53:17.074 INFO:teuthology.orchestra.run.smithi114.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0126066 s, 832 MB/s 2024-04-12T21:53:17.074 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping successful for: 2024-04-12T21:53:17.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:17 smithi114 ceph-mon[25759]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-04-12T21:53:17.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:17 smithi114 ceph-mon[25759]: osdmap e24: 4 total, 3 up, 4 in 2024-04-12T21:53:17.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:17 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T21:53:17.646 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:17 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T21:53:17.667 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- ceph orch daemon add osd smithi114:vg_nvme/lv_4 2024-04-12T21:53:17.763 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:17 smithi097 ceph-mon[24835]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-04-12T21:53:17.763 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:17 smithi097 ceph-mon[24835]: osdmap e24: 4 total, 3 up, 4 in 2024-04-12T21:53:17.763 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:17 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T21:53:17.763 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:17 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T21:53:17.764 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:17 smithi097 ceph-mon[29620]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-04-12T21:53:17.764 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:17 smithi097 ceph-mon[29620]: osdmap e24: 4 total, 3 up, 4 in 2024-04-12T21:53:17.764 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:17 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T21:53:17.764 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:17 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T21:53:18.869 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:18 smithi114 ceph-mon[25759]: purged_snaps scrub starts 2024-04-12T21:53:18.869 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:18 smithi114 ceph-mon[25759]: purged_snaps scrub ok 2024-04-12T21:53:18.869 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:18 smithi114 ceph-mon[25759]: osd.3 [v2:172.21.15.97:6826/75554646,v1:172.21.15.97:6827/75554646] boot 2024-04-12T21:53:18.869 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:18 smithi114 ceph-mon[25759]: osdmap e25: 4 total, 4 up, 4 in 2024-04-12T21:53:18.870 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:18 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T21:53:18.870 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:18 smithi114 ceph-mon[25759]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:53:18.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:18 smithi097 ceph-mon[29620]: purged_snaps scrub starts 2024-04-12T21:53:18.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:18 smithi097 ceph-mon[29620]: purged_snaps scrub ok 2024-04-12T21:53:18.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:18 smithi097 ceph-mon[29620]: osd.3 [v2:172.21.15.97:6826/75554646,v1:172.21.15.97:6827/75554646] boot 2024-04-12T21:53:18.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:18 smithi097 ceph-mon[29620]: osdmap e25: 4 total, 4 up, 4 in 2024-04-12T21:53:18.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:18 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T21:53:18.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:18 smithi097 ceph-mon[29620]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:53:18.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:18 smithi097 ceph-mon[24835]: purged_snaps scrub starts 2024-04-12T21:53:18.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:18 smithi097 ceph-mon[24835]: purged_snaps scrub ok 2024-04-12T21:53:18.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:18 smithi097 ceph-mon[24835]: osd.3 [v2:172.21.15.97:6826/75554646,v1:172.21.15.97:6827/75554646] boot 2024-04-12T21:53:18.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:18 smithi097 ceph-mon[24835]: osdmap e25: 4 total, 4 up, 4 in 2024-04-12T21:53:18.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:18 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T21:53:18.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:18 smithi097 ceph-mon[24835]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:53:19.831 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:19 smithi114 ceph-mon[25759]: osdmap e26: 4 total, 4 up, 4 in 2024-04-12T21:53:19.831 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:19 smithi114 ceph-mon[25759]: Detected new or changed devices on smithi097 2024-04-12T21:53:19.832 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:19 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:53:19.832 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:19 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:53:19.832 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:19 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:53:19.832 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:19 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:53:19.832 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:19 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:53:19.832 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:19 smithi114 ceph-mon[25759]: Adjusting osd_memory_target on smithi097 to 4016M 2024-04-12T21:53:19.832 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:19 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:53:19.832 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:19 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:53:19.832 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:19 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:53:19.832 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:19 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:53:19.832 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:19 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:53:19.832 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:19 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:53:19.832 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:19 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:53:19.832 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:19 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:53:19.833 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:19 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:53:19.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:19 smithi097 ceph-mon[24835]: osdmap e26: 4 total, 4 up, 4 in 2024-04-12T21:53:19.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:19 smithi097 ceph-mon[24835]: Detected new or changed devices on smithi097 2024-04-12T21:53:19.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:19 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:53:19.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:19 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:53:19.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:19 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:53:19.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:19 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:53:19.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:19 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:53:19.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:19 smithi097 ceph-mon[24835]: Adjusting osd_memory_target on smithi097 to 4016M 2024-04-12T21:53:19.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:19 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:53:19.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:19 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:53:19.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:19 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:53:19.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:19 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:53:19.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:19 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:53:19.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:19 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:53:19.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:19 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:53:19.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:19 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:53:19.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:19 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:53:19.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:19 smithi097 ceph-mon[29620]: osdmap e26: 4 total, 4 up, 4 in 2024-04-12T21:53:19.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:19 smithi097 ceph-mon[29620]: Detected new or changed devices on smithi097 2024-04-12T21:53:19.913 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:19 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:53:19.913 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:19 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:53:19.913 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:19 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:53:19.913 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:19 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:53:19.913 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:19 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:53:19.913 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:19 smithi097 ceph-mon[29620]: Adjusting osd_memory_target on smithi097 to 4016M 2024-04-12T21:53:19.913 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:19 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:53:19.913 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:19 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:53:19.913 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:19 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:53:19.913 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:19 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:53:19.913 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:19 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:53:19.913 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:19 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:53:19.913 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:19 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:53:19.913 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:19 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:53:19.914 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:19 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:53:20.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:20 smithi114 ceph-mon[25759]: from='client.24188 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:53:20.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:20 smithi114 ceph-mon[25759]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-12T21:53:20.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:20 smithi097 ceph-mon[24835]: from='client.24188 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:53:20.913 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:20 smithi097 ceph-mon[24835]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-12T21:53:20.913 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:20 smithi097 ceph-mon[29620]: from='client.24188 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:53:20.913 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:20 smithi097 ceph-mon[29620]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-12T21:53:21.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:21 smithi114 ceph-mon[25759]: from='client.? 172.21.15.114:0/652773296' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "02f49773-bcfc-499c-9174-d9cb5991c789"}]: dispatch 2024-04-12T21:53:21.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:21 smithi114 ceph-mon[25759]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "02f49773-bcfc-499c-9174-d9cb5991c789"}]: dispatch 2024-04-12T21:53:21.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:21 smithi114 ceph-mon[25759]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "02f49773-bcfc-499c-9174-d9cb5991c789"}]': finished 2024-04-12T21:53:21.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:21 smithi114 ceph-mon[25759]: osdmap e27: 5 total, 4 up, 5 in 2024-04-12T21:53:21.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:21 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T21:53:22.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:21 smithi097 ceph-mon[24835]: from='client.? 172.21.15.114:0/652773296' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "02f49773-bcfc-499c-9174-d9cb5991c789"}]: dispatch 2024-04-12T21:53:22.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:21 smithi097 ceph-mon[24835]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "02f49773-bcfc-499c-9174-d9cb5991c789"}]: dispatch 2024-04-12T21:53:22.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:21 smithi097 ceph-mon[24835]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "02f49773-bcfc-499c-9174-d9cb5991c789"}]': finished 2024-04-12T21:53:22.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:21 smithi097 ceph-mon[24835]: osdmap e27: 5 total, 4 up, 5 in 2024-04-12T21:53:22.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:21 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T21:53:22.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:21 smithi097 ceph-mon[29620]: from='client.? 172.21.15.114:0/652773296' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "02f49773-bcfc-499c-9174-d9cb5991c789"}]: dispatch 2024-04-12T21:53:22.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:21 smithi097 ceph-mon[29620]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "02f49773-bcfc-499c-9174-d9cb5991c789"}]: dispatch 2024-04-12T21:53:22.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:21 smithi097 ceph-mon[29620]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "02f49773-bcfc-499c-9174-d9cb5991c789"}]': finished 2024-04-12T21:53:22.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:21 smithi097 ceph-mon[29620]: osdmap e27: 5 total, 4 up, 5 in 2024-04-12T21:53:22.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:21 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T21:53:22.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:22 smithi114 ceph-mon[25759]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-12T21:53:22.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:22 smithi114 ceph-mon[25759]: from='client.? 172.21.15.114:0/2024428371' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:53:23.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:22 smithi097 ceph-mon[24835]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-12T21:53:23.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:22 smithi097 ceph-mon[24835]: from='client.? 172.21.15.114:0/2024428371' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:53:23.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:22 smithi097 ceph-mon[29620]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-12T21:53:23.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:22 smithi097 ceph-mon[29620]: from='client.? 172.21.15.114:0/2024428371' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:53:25.624 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:25 smithi114 ceph-mon[25759]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-12T21:53:25.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:25 smithi097 ceph-mon[24835]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-12T21:53:25.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:25 smithi097 ceph-mon[29620]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-12T21:53:27.456 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:27 smithi114 ceph-mon[25759]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-12T21:53:27.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:27 smithi097 ceph-mon[24835]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-12T21:53:27.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:27 smithi097 ceph-mon[29620]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-12T21:53:28.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:28 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-12T21:53:28.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:28 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:53:28.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:28 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-12T21:53:28.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:28 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:53:28.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:28 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-12T21:53:28.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:28 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:53:29.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:29 smithi097 ceph-mon[24835]: Deploying daemon osd.4 on smithi114 2024-04-12T21:53:29.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:29 smithi097 ceph-mon[24835]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-12T21:53:29.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:29 smithi097 ceph-mon[29620]: Deploying daemon osd.4 on smithi114 2024-04-12T21:53:29.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:29 smithi097 ceph-mon[29620]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-12T21:53:29.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:29 smithi114 ceph-mon[25759]: Deploying daemon osd.4 on smithi114 2024-04-12T21:53:29.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:29 smithi114 ceph-mon[25759]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-12T21:53:31.467 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:31 smithi114 ceph-mon[25759]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-12T21:53:31.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:31 smithi097 ceph-mon[24835]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-12T21:53:31.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:31 smithi097 ceph-mon[29620]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-12T21:53:33.601 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:33 smithi114 ceph-mon[25759]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-12T21:53:33.602 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:33 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:53:33.602 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:33 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:53:33.602 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:33 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:53:33.602 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:33 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:53:33.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:33 smithi097 ceph-mon[24835]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-12T21:53:33.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:33 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:53:33.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:33 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:53:33.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:33 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:53:33.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:33 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:53:33.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:33 smithi097 ceph-mon[29620]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-12T21:53:33.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:33 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:53:33.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:33 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:53:33.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:33 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:53:33.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:33 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:53:34.998 INFO:teuthology.orchestra.run.smithi114.stdout:Created osd(s) 4 on host 'smithi114' 2024-04-12T21:53:35.483 DEBUG:teuthology.orchestra.run.smithi114:osd.4> sudo journalctl -f -n 0 -u ceph-80bf7680-f916-11ee-bc8d-c7b262605968@osd.4.service 2024-04-12T21:53:35.486 INFO:tasks.cephadm:Deploying osd.5 on smithi114 with /dev/vg_nvme/lv_3... 2024-04-12T21:53:35.486 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-12T21:53:35.550 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:35 smithi114 ceph-mon[25759]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-12T21:53:35.550 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:35 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:53:35.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:35 smithi097 ceph-mon[24835]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-12T21:53:35.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:35 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:53:35.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:35 smithi097 ceph-mon[29620]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-12T21:53:35.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:35 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:53:36.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:36 smithi114 ceph-mon[25759]: from='osd.4 [v2:172.21.15.114:6800/2190104221,v1:172.21.15.114:6801/2190104221]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-12T21:53:36.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:36 smithi114 ceph-mon[25759]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-12T21:53:36.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:36 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:53:36.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:36 smithi097 ceph-mon[29620]: from='osd.4 [v2:172.21.15.114:6800/2190104221,v1:172.21.15.114:6801/2190104221]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-12T21:53:36.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:36 smithi097 ceph-mon[29620]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-12T21:53:36.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:36 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:53:36.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:36 smithi097 ceph-mon[24835]: from='osd.4 [v2:172.21.15.114:6800/2190104221,v1:172.21.15.114:6801/2190104221]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-12T21:53:36.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:36 smithi097 ceph-mon[24835]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-12T21:53:36.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:36 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:53:37.576 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-12T21:53:37.577 INFO:teuthology.orchestra.run.smithi114.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-12T21:53:37.577 INFO:teuthology.orchestra.run.smithi114.stderr: stderr: 10+0 records in 2024-04-12T21:53:37.577 INFO:teuthology.orchestra.run.smithi114.stderr:10+0 records out 2024-04-12T21:53:37.577 INFO:teuthology.orchestra.run.smithi114.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0127353 s, 823 MB/s 2024-04-12T21:53:37.577 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping successful for: 2024-04-12T21:53:37.644 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:37 smithi114 ceph-mon[25759]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-12T21:53:37.644 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:37 smithi114 ceph-mon[25759]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-12T21:53:37.644 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:37 smithi114 ceph-mon[25759]: osdmap e28: 5 total, 4 up, 5 in 2024-04-12T21:53:37.644 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:37 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T21:53:37.644 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:37 smithi114 ceph-mon[25759]: from='osd.4 [v2:172.21.15.114:6800/2190104221,v1:172.21.15.114:6801/2190104221]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-12T21:53:37.644 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:37 smithi114 ceph-mon[25759]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-12T21:53:37.644 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 21:53:37 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-4[29183]: 2024-04-12T21:53:37.365+0000 7f9e809ae700 -1 osd.4 0 waiting for initial osdmap 2024-04-12T21:53:37.644 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 21:53:37 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-4[29183]: 2024-04-12T21:53:37.370+0000 7f9e7ab44700 -1 osd.4 29 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T21:53:37.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:37 smithi097 ceph-mon[24835]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-12T21:53:37.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:37 smithi097 ceph-mon[24835]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-12T21:53:37.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:37 smithi097 ceph-mon[24835]: osdmap e28: 5 total, 4 up, 5 in 2024-04-12T21:53:37.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:37 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T21:53:37.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:37 smithi097 ceph-mon[24835]: from='osd.4 [v2:172.21.15.114:6800/2190104221,v1:172.21.15.114:6801/2190104221]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-12T21:53:37.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:37 smithi097 ceph-mon[24835]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-12T21:53:37.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:37 smithi097 ceph-mon[29620]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-12T21:53:37.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:37 smithi097 ceph-mon[29620]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-12T21:53:37.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:37 smithi097 ceph-mon[29620]: osdmap e28: 5 total, 4 up, 5 in 2024-04-12T21:53:37.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:37 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T21:53:37.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:37 smithi097 ceph-mon[29620]: from='osd.4 [v2:172.21.15.114:6800/2190104221,v1:172.21.15.114:6801/2190104221]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-12T21:53:37.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:37 smithi097 ceph-mon[29620]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-12T21:53:38.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:38 smithi097 ceph-mon[29620]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-04-12T21:53:38.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:38 smithi097 ceph-mon[29620]: osdmap e29: 5 total, 4 up, 5 in 2024-04-12T21:53:38.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:38 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T21:53:38.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:38 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T21:53:38.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:38 smithi097 ceph-mon[29620]: osd.4 [v2:172.21.15.114:6800/2190104221,v1:172.21.15.114:6801/2190104221] boot 2024-04-12T21:53:38.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:38 smithi097 ceph-mon[29620]: osdmap e30: 5 total, 5 up, 5 in 2024-04-12T21:53:38.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:38 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T21:53:38.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:38 smithi097 ceph-mon[24835]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-04-12T21:53:38.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:38 smithi097 ceph-mon[24835]: osdmap e29: 5 total, 4 up, 5 in 2024-04-12T21:53:38.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:38 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T21:53:38.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:38 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T21:53:38.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:38 smithi097 ceph-mon[24835]: osd.4 [v2:172.21.15.114:6800/2190104221,v1:172.21.15.114:6801/2190104221] boot 2024-04-12T21:53:38.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:38 smithi097 ceph-mon[24835]: osdmap e30: 5 total, 5 up, 5 in 2024-04-12T21:53:38.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:38 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T21:53:38.683 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:38 smithi114 ceph-mon[25759]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-04-12T21:53:38.683 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:38 smithi114 ceph-mon[25759]: osdmap e29: 5 total, 4 up, 5 in 2024-04-12T21:53:38.683 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:38 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T21:53:38.683 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:38 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T21:53:38.683 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:38 smithi114 ceph-mon[25759]: osd.4 [v2:172.21.15.114:6800/2190104221,v1:172.21.15.114:6801/2190104221] boot 2024-04-12T21:53:38.683 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:38 smithi114 ceph-mon[25759]: osdmap e30: 5 total, 5 up, 5 in 2024-04-12T21:53:38.683 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:38 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T21:53:39.302 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- ceph orch daemon add osd smithi114:vg_nvme/lv_3 2024-04-12T21:53:39.638 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:39 smithi114 ceph-mon[25759]: purged_snaps scrub starts 2024-04-12T21:53:39.638 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:39 smithi114 ceph-mon[25759]: purged_snaps scrub ok 2024-04-12T21:53:39.638 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:39 smithi114 ceph-mon[25759]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-12T21:53:39.638 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:39 smithi114 ceph-mon[25759]: osdmap e31: 5 total, 5 up, 5 in 2024-04-12T21:53:39.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:39 smithi097 ceph-mon[29620]: purged_snaps scrub starts 2024-04-12T21:53:39.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:39 smithi097 ceph-mon[29620]: purged_snaps scrub ok 2024-04-12T21:53:39.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:39 smithi097 ceph-mon[29620]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-12T21:53:39.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:39 smithi097 ceph-mon[29620]: osdmap e31: 5 total, 5 up, 5 in 2024-04-12T21:53:39.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:39 smithi097 ceph-mon[24835]: purged_snaps scrub starts 2024-04-12T21:53:39.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:39 smithi097 ceph-mon[24835]: purged_snaps scrub ok 2024-04-12T21:53:39.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:39 smithi097 ceph-mon[24835]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-12T21:53:39.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:39 smithi097 ceph-mon[24835]: osdmap e31: 5 total, 5 up, 5 in 2024-04-12T21:53:41.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:41 smithi114 ceph-mon[25759]: pgmap v109: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-12T21:53:41.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:41 smithi114 ceph-mon[25759]: osdmap e32: 5 total, 5 up, 5 in 2024-04-12T21:53:41.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:41 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:53:41.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:41 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:53:41.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:41 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:53:41.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:41 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:53:41.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:41 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:53:41.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:41 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:53:41.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:41 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:53:41.495 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:41 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:53:41.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:41 smithi097 ceph-mon[29620]: pgmap v109: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-12T21:53:41.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:41 smithi097 ceph-mon[29620]: osdmap e32: 5 total, 5 up, 5 in 2024-04-12T21:53:41.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:41 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:53:41.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:41 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:53:41.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:41 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:53:41.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:41 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:53:41.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:41 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:53:41.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:41 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:53:41.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:41 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:53:41.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:41 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:53:41.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:41 smithi097 ceph-mon[24835]: pgmap v109: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-12T21:53:41.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:41 smithi097 ceph-mon[24835]: osdmap e32: 5 total, 5 up, 5 in 2024-04-12T21:53:41.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:41 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:53:41.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:41 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:53:41.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:41 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:53:41.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:41 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:53:41.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:41 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:53:41.663 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:41 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:53:41.663 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:41 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:53:41.663 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:41 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:53:42.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:42 smithi097 ceph-mon[24835]: Detected new or changed devices on smithi114 2024-04-12T21:53:42.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:42 smithi097 ceph-mon[24835]: Adjusting osd_memory_target on smithi114 to 17088M 2024-04-12T21:53:42.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:42 smithi097 ceph-mon[24835]: osdmap e33: 5 total, 5 up, 5 in 2024-04-12T21:53:42.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:42 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:53:42.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:42 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:53:42.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:42 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:53:42.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:42 smithi097 ceph-mon[29620]: Detected new or changed devices on smithi114 2024-04-12T21:53:42.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:42 smithi097 ceph-mon[29620]: Adjusting osd_memory_target on smithi114 to 17088M 2024-04-12T21:53:42.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:42 smithi097 ceph-mon[29620]: osdmap e33: 5 total, 5 up, 5 in 2024-04-12T21:53:42.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:42 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:53:42.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:42 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:53:42.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:42 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:53:42.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:42 smithi114 ceph-mon[25759]: Detected new or changed devices on smithi114 2024-04-12T21:53:42.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:42 smithi114 ceph-mon[25759]: Adjusting osd_memory_target on smithi114 to 17088M 2024-04-12T21:53:42.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:42 smithi114 ceph-mon[25759]: osdmap e33: 5 total, 5 up, 5 in 2024-04-12T21:53:42.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:42 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:53:42.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:42 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:53:42.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:42 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:53:43.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:43 smithi097 ceph-mon[24835]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:53:43.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:43 smithi097 ceph-mon[24835]: pgmap v112: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-12T21:53:43.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:43 smithi097 ceph-mon[24835]: osdmap e34: 5 total, 5 up, 5 in 2024-04-12T21:53:43.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:43 smithi097 ceph-mon[29620]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:53:43.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:43 smithi097 ceph-mon[29620]: pgmap v112: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-12T21:53:43.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:43 smithi097 ceph-mon[29620]: osdmap e34: 5 total, 5 up, 5 in 2024-04-12T21:53:43.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:43 smithi114 ceph-mon[25759]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:53:43.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:43 smithi114 ceph-mon[25759]: pgmap v112: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-12T21:53:43.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:43 smithi114 ceph-mon[25759]: osdmap e34: 5 total, 5 up, 5 in 2024-04-12T21:53:44.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:44 smithi097 ceph-mon[24835]: from='client.? 172.21.15.114:0/2768344276' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "997405e6-e6f2-4c40-b81c-7848c56c062c"}]: dispatch 2024-04-12T21:53:44.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:44 smithi097 ceph-mon[24835]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "997405e6-e6f2-4c40-b81c-7848c56c062c"}]: dispatch 2024-04-12T21:53:44.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:44 smithi097 ceph-mon[24835]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "997405e6-e6f2-4c40-b81c-7848c56c062c"}]': finished 2024-04-12T21:53:44.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:44 smithi097 ceph-mon[24835]: osdmap e35: 6 total, 5 up, 6 in 2024-04-12T21:53:44.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:44 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T21:53:44.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:44 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T21:53:44.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:44 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T21:53:44.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:44 smithi097 ceph-mon[24835]: from='client.? 172.21.15.114:0/378914704' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:53:44.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:44 smithi097 ceph-mon[29620]: from='client.? 172.21.15.114:0/2768344276' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "997405e6-e6f2-4c40-b81c-7848c56c062c"}]: dispatch 2024-04-12T21:53:44.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:44 smithi097 ceph-mon[29620]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "997405e6-e6f2-4c40-b81c-7848c56c062c"}]: dispatch 2024-04-12T21:53:44.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:44 smithi097 ceph-mon[29620]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "997405e6-e6f2-4c40-b81c-7848c56c062c"}]': finished 2024-04-12T21:53:44.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:44 smithi097 ceph-mon[29620]: osdmap e35: 6 total, 5 up, 6 in 2024-04-12T21:53:44.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:44 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T21:53:44.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:44 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T21:53:44.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:44 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T21:53:44.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:44 smithi097 ceph-mon[29620]: from='client.? 172.21.15.114:0/378914704' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:53:44.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:44 smithi114 ceph-mon[25759]: from='client.? 172.21.15.114:0/2768344276' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "997405e6-e6f2-4c40-b81c-7848c56c062c"}]: dispatch 2024-04-12T21:53:44.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:44 smithi114 ceph-mon[25759]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "997405e6-e6f2-4c40-b81c-7848c56c062c"}]: dispatch 2024-04-12T21:53:44.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:44 smithi114 ceph-mon[25759]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "997405e6-e6f2-4c40-b81c-7848c56c062c"}]': finished 2024-04-12T21:53:44.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:44 smithi114 ceph-mon[25759]: osdmap e35: 6 total, 5 up, 6 in 2024-04-12T21:53:44.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:44 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T21:53:44.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:44 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T21:53:44.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:44 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T21:53:44.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:44 smithi114 ceph-mon[25759]: from='client.? 172.21.15.114:0/378914704' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:53:45.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:45 smithi097 ceph-mon[24835]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-12T21:53:45.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:45 smithi097 ceph-mon[29620]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-12T21:53:45.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:45 smithi114 ceph-mon[25759]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-12T21:53:47.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:47 smithi097 ceph-mon[24835]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-04-12T21:53:47.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:47 smithi097 ceph-mon[29620]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-04-12T21:53:47.741 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:47 smithi114 ceph-mon[25759]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-04-12T21:53:49.633 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:49 smithi114 ceph-mon[25759]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-04-12T21:53:49.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:49 smithi097 ceph-mon[24835]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-04-12T21:53:49.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:49 smithi097 ceph-mon[29620]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-04-12T21:53:50.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:50 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-12T21:53:50.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:50 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:53:50.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:50 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-12T21:53:50.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:50 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:53:50.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:50 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-12T21:53:50.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:50 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:53:51.572 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:51 smithi114 ceph-mon[25759]: Deploying daemon osd.5 on smithi114 2024-04-12T21:53:51.625 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:51 smithi114 ceph-mon[25759]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-12T21:53:51.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:51 smithi097 ceph-mon[24835]: Deploying daemon osd.5 on smithi114 2024-04-12T21:53:51.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:51 smithi097 ceph-mon[24835]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-12T21:53:51.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:51 smithi097 ceph-mon[29620]: Deploying daemon osd.5 on smithi114 2024-04-12T21:53:51.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:51 smithi097 ceph-mon[29620]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-12T21:53:53.635 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:53 smithi114 ceph-mon[25759]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-04-12T21:53:53.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:53 smithi097 ceph-mon[24835]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-04-12T21:53:53.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:53 smithi097 ceph-mon[29620]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-04-12T21:53:55.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:55 smithi097 ceph-mon[24835]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-04-12T21:53:55.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:55 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:53:55.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:55 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:53:55.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:55 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:53:55.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:55 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:53:55.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:55 smithi097 ceph-mon[29620]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-04-12T21:53:55.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:55 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:53:55.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:55 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:53:55.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:55 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:53:55.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:55 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:53:55.695 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:55 smithi114 ceph-mon[25759]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-04-12T21:53:55.695 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:55 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:53:55.695 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:55 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:53:55.695 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:55 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:53:55.695 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:55 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:53:56.917 INFO:teuthology.orchestra.run.smithi114.stdout:Created osd(s) 5 on host 'smithi114' 2024-04-12T21:53:57.462 DEBUG:teuthology.orchestra.run.smithi114:osd.5> sudo journalctl -f -n 0 -u ceph-80bf7680-f916-11ee-bc8d-c7b262605968@osd.5.service 2024-04-12T21:53:57.464 INFO:tasks.cephadm:Deploying osd.6 on smithi114 with /dev/vg_nvme/lv_2... 2024-04-12T21:53:57.465 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-12T21:53:57.488 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:57 smithi114 ceph-mon[25759]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-04-12T21:53:57.488 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:57 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:53:57.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:57 smithi097 ceph-mon[24835]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-04-12T21:53:57.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:57 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:53:57.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:57 smithi097 ceph-mon[29620]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-04-12T21:53:57.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:57 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:53:57.744 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 21:53:57 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5[32059]: 2024-04-12T21:53:57.553+0000 7f6ce14b73c0 -1 osd.5 0 log_to_monitors true 2024-04-12T21:53:58.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:58 smithi097 ceph-mon[24835]: from='osd.5 [v2:172.21.15.114:6808/515183706,v1:172.21.15.114:6809/515183706]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-12T21:53:58.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:58 smithi097 ceph-mon[24835]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-12T21:53:58.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:58 smithi097 ceph-mon[29620]: from='osd.5 [v2:172.21.15.114:6808/515183706,v1:172.21.15.114:6809/515183706]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-12T21:53:58.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:58 smithi097 ceph-mon[29620]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-12T21:53:58.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:58 smithi114 ceph-mon[25759]: from='osd.5 [v2:172.21.15.114:6808/515183706,v1:172.21.15.114:6809/515183706]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-12T21:53:58.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:58 smithi114 ceph-mon[25759]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-12T21:53:59.528 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-12T21:53:59.529 INFO:teuthology.orchestra.run.smithi114.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-12T21:53:59.529 INFO:teuthology.orchestra.run.smithi114.stderr: stderr: 10+0 records in 2024-04-12T21:53:59.529 INFO:teuthology.orchestra.run.smithi114.stderr:10+0 records out 2024-04-12T21:53:59.529 INFO:teuthology.orchestra.run.smithi114.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0127282 s, 824 MB/s 2024-04-12T21:53:59.529 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping successful for: 2024-04-12T21:53:59.744 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 21:53:59 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5[32059]: 2024-04-12T21:53:59.418+0000 7f6cd7eba700 -1 osd.5 0 waiting for initial osdmap 2024-04-12T21:53:59.744 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 21:53:59 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5[32059]: 2024-04-12T21:53:59.423+0000 7f6cd3052700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T21:53:59.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:59 smithi114 ceph-mon[25759]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-12T21:53:59.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:59 smithi114 ceph-mon[25759]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-12T21:53:59.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:59 smithi114 ceph-mon[25759]: osdmap e36: 6 total, 5 up, 6 in 2024-04-12T21:53:59.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:59 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T21:53:59.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:59 smithi114 ceph-mon[25759]: from='osd.5 [v2:172.21.15.114:6808/515183706,v1:172.21.15.114:6809/515183706]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-12T21:53:59.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:59 smithi114 ceph-mon[25759]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-12T21:53:59.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:53:59 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:53:59.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:59 smithi097 ceph-mon[24835]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-12T21:53:59.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:59 smithi097 ceph-mon[24835]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-12T21:53:59.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:59 smithi097 ceph-mon[24835]: osdmap e36: 6 total, 5 up, 6 in 2024-04-12T21:53:59.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:59 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T21:53:59.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:59 smithi097 ceph-mon[24835]: from='osd.5 [v2:172.21.15.114:6808/515183706,v1:172.21.15.114:6809/515183706]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-12T21:53:59.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:59 smithi097 ceph-mon[24835]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-12T21:53:59.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:53:59 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:53:59.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:59 smithi097 ceph-mon[29620]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-12T21:53:59.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:59 smithi097 ceph-mon[29620]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-12T21:53:59.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:59 smithi097 ceph-mon[29620]: osdmap e36: 6 total, 5 up, 6 in 2024-04-12T21:53:59.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:59 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T21:53:59.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:59 smithi097 ceph-mon[29620]: from='osd.5 [v2:172.21.15.114:6808/515183706,v1:172.21.15.114:6809/515183706]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-12T21:53:59.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:59 smithi097 ceph-mon[29620]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-12T21:53:59.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:53:59 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:00.134 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- ceph orch daemon add osd smithi114:vg_nvme/lv_2 2024-04-12T21:54:00.540 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:00 smithi114 ceph-mon[25759]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-04-12T21:54:00.540 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:00 smithi114 ceph-mon[25759]: osdmap e37: 6 total, 5 up, 6 in 2024-04-12T21:54:00.540 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:00 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T21:54:00.540 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:00 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T21:54:00.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:00 smithi097 ceph-mon[24835]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-04-12T21:54:00.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:00 smithi097 ceph-mon[24835]: osdmap e37: 6 total, 5 up, 6 in 2024-04-12T21:54:00.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:00 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T21:54:00.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:00 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T21:54:00.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:00 smithi097 ceph-mon[29620]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-04-12T21:54:00.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:00 smithi097 ceph-mon[29620]: osdmap e37: 6 total, 5 up, 6 in 2024-04-12T21:54:00.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:00 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T21:54:00.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:00 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T21:54:01.587 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:01 smithi114 ceph-mon[25759]: purged_snaps scrub starts 2024-04-12T21:54:01.587 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:01 smithi114 ceph-mon[25759]: purged_snaps scrub ok 2024-04-12T21:54:01.587 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:01 smithi114 ceph-mon[25759]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-12T21:54:01.588 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:01 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T21:54:01.588 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:01 smithi114 ceph-mon[25759]: osd.5 [v2:172.21.15.114:6808/515183706,v1:172.21.15.114:6809/515183706] boot 2024-04-12T21:54:01.588 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:01 smithi114 ceph-mon[25759]: osdmap e38: 6 total, 6 up, 6 in 2024-04-12T21:54:01.588 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:01 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T21:54:01.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:01 smithi097 ceph-mon[24835]: purged_snaps scrub starts 2024-04-12T21:54:01.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:01 smithi097 ceph-mon[24835]: purged_snaps scrub ok 2024-04-12T21:54:01.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:01 smithi097 ceph-mon[24835]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-12T21:54:01.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:01 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T21:54:01.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:01 smithi097 ceph-mon[24835]: osd.5 [v2:172.21.15.114:6808/515183706,v1:172.21.15.114:6809/515183706] boot 2024-04-12T21:54:01.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:01 smithi097 ceph-mon[24835]: osdmap e38: 6 total, 6 up, 6 in 2024-04-12T21:54:01.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:01 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T21:54:01.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:01 smithi097 ceph-mon[29620]: purged_snaps scrub starts 2024-04-12T21:54:01.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:01 smithi097 ceph-mon[29620]: purged_snaps scrub ok 2024-04-12T21:54:01.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:01 smithi097 ceph-mon[29620]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-12T21:54:01.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:01 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T21:54:01.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:01 smithi097 ceph-mon[29620]: osd.5 [v2:172.21.15.114:6808/515183706,v1:172.21.15.114:6809/515183706] boot 2024-04-12T21:54:01.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:01 smithi097 ceph-mon[29620]: osdmap e38: 6 total, 6 up, 6 in 2024-04-12T21:54:01.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:01 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T21:54:02.644 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:02 smithi114 ceph-mon[25759]: osdmap e39: 6 total, 6 up, 6 in 2024-04-12T21:54:02.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:02 smithi114 ceph-mon[25759]: pgmap v128: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-12T21:54:02.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:02 smithi114 ceph-mon[25759]: osdmap e40: 6 total, 6 up, 6 in 2024-04-12T21:54:02.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:02 smithi097 ceph-mon[24835]: osdmap e39: 6 total, 6 up, 6 in 2024-04-12T21:54:02.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:02 smithi097 ceph-mon[24835]: pgmap v128: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-12T21:54:02.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:02 smithi097 ceph-mon[24835]: osdmap e40: 6 total, 6 up, 6 in 2024-04-12T21:54:02.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:02 smithi097 ceph-mon[29620]: osdmap e39: 6 total, 6 up, 6 in 2024-04-12T21:54:02.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:02 smithi097 ceph-mon[29620]: pgmap v128: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-12T21:54:02.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:02 smithi097 ceph-mon[29620]: osdmap e40: 6 total, 6 up, 6 in 2024-04-12T21:54:03.483 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:03 smithi114 ceph-mon[25759]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-12T21:54:03.483 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:03 smithi114 ceph-mon[25759]: osdmap e41: 6 total, 6 up, 6 in 2024-04-12T21:54:03.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:03 smithi097 ceph-mon[24835]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-12T21:54:03.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:03 smithi097 ceph-mon[24835]: osdmap e41: 6 total, 6 up, 6 in 2024-04-12T21:54:03.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:03 smithi097 ceph-mon[29620]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-12T21:54:03.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:03 smithi097 ceph-mon[29620]: osdmap e41: 6 total, 6 up, 6 in 2024-04-12T21:54:04.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:04 smithi114 ceph-mon[25759]: from='client.24242 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:54:04.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:04 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:54:04.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:04 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:54:04.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:04 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:54:04.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:04 smithi114 ceph-mon[25759]: Detected new or changed devices on smithi114 2024-04-12T21:54:04.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:04 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:04.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:04 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:54:04.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:04 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:54:04.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:04 smithi114 ceph-mon[25759]: Adjusting osd_memory_target on smithi114 to 8544M 2024-04-12T21:54:04.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:04 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:04.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:04 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:04.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:04 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:54:04.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:04 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:54:04.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:04 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:54:04.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:04 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:04.746 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:04 smithi114 ceph-mon[25759]: pgmap v131: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-12T21:54:04.746 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:04 smithi114 ceph-mon[25759]: osdmap e42: 6 total, 6 up, 6 in 2024-04-12T21:54:04.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:04 smithi097 ceph-mon[24835]: from='client.24242 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:54:04.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:04 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:54:04.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:04 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:54:04.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:04 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:54:04.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:04 smithi097 ceph-mon[24835]: Detected new or changed devices on smithi114 2024-04-12T21:54:04.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:04 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:04.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:04 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:54:04.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:04 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:54:04.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:04 smithi097 ceph-mon[24835]: Adjusting osd_memory_target on smithi114 to 8544M 2024-04-12T21:54:04.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:04 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:04.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:04 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:04.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:04 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:54:04.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:04 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:54:04.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:04 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:54:04.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:04 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:04.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:04 smithi097 ceph-mon[24835]: pgmap v131: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-12T21:54:04.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:04 smithi097 ceph-mon[24835]: osdmap e42: 6 total, 6 up, 6 in 2024-04-12T21:54:04.913 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:04 smithi097 ceph-mon[29620]: from='client.24242 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:54:04.913 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:04 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:54:04.913 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:04 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:54:04.913 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:04 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:54:04.913 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:04 smithi097 ceph-mon[29620]: Detected new or changed devices on smithi114 2024-04-12T21:54:04.913 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:04 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:04.913 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:04 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:54:04.913 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:04 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:54:04.913 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:04 smithi097 ceph-mon[29620]: Adjusting osd_memory_target on smithi114 to 8544M 2024-04-12T21:54:04.913 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:04 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:04.913 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:04 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:04.913 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:04 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:54:04.913 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:04 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:54:04.913 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:04 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:54:04.914 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:04 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:04.914 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:04 smithi097 ceph-mon[29620]: pgmap v131: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-12T21:54:04.914 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:04 smithi097 ceph-mon[29620]: osdmap e42: 6 total, 6 up, 6 in 2024-04-12T21:54:05.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:05 smithi114 ceph-mon[25759]: from='client.? 172.21.15.114:0/1863080911' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "68b36dad-0aff-40a4-b8f7-8c1dcf42a5c8"}]: dispatch 2024-04-12T21:54:05.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:05 smithi114 ceph-mon[25759]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "68b36dad-0aff-40a4-b8f7-8c1dcf42a5c8"}]: dispatch 2024-04-12T21:54:05.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:05 smithi114 ceph-mon[25759]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "68b36dad-0aff-40a4-b8f7-8c1dcf42a5c8"}]': finished 2024-04-12T21:54:05.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:05 smithi114 ceph-mon[25759]: osdmap e43: 7 total, 6 up, 7 in 2024-04-12T21:54:05.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:05 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-12T21:54:06.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:05 smithi097 ceph-mon[24835]: from='client.? 172.21.15.114:0/1863080911' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "68b36dad-0aff-40a4-b8f7-8c1dcf42a5c8"}]: dispatch 2024-04-12T21:54:06.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:05 smithi097 ceph-mon[24835]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "68b36dad-0aff-40a4-b8f7-8c1dcf42a5c8"}]: dispatch 2024-04-12T21:54:06.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:05 smithi097 ceph-mon[24835]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "68b36dad-0aff-40a4-b8f7-8c1dcf42a5c8"}]': finished 2024-04-12T21:54:06.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:05 smithi097 ceph-mon[24835]: osdmap e43: 7 total, 6 up, 7 in 2024-04-12T21:54:06.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:05 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-12T21:54:06.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:05 smithi097 ceph-mon[29620]: from='client.? 172.21.15.114:0/1863080911' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "68b36dad-0aff-40a4-b8f7-8c1dcf42a5c8"}]: dispatch 2024-04-12T21:54:06.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:05 smithi097 ceph-mon[29620]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "68b36dad-0aff-40a4-b8f7-8c1dcf42a5c8"}]: dispatch 2024-04-12T21:54:06.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:05 smithi097 ceph-mon[29620]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "68b36dad-0aff-40a4-b8f7-8c1dcf42a5c8"}]': finished 2024-04-12T21:54:06.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:05 smithi097 ceph-mon[29620]: osdmap e43: 7 total, 6 up, 7 in 2024-04-12T21:54:06.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:05 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-12T21:54:06.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:06 smithi114 ceph-mon[25759]: pgmap v134: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-12T21:54:06.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:06 smithi114 ceph-mon[25759]: from='client.? 172.21.15.114:0/2009207526' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:54:07.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:06 smithi097 ceph-mon[24835]: pgmap v134: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-12T21:54:07.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:06 smithi097 ceph-mon[24835]: from='client.? 172.21.15.114:0/2009207526' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:54:07.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:06 smithi097 ceph-mon[29620]: pgmap v134: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-12T21:54:07.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:06 smithi097 ceph-mon[29620]: from='client.? 172.21.15.114:0/2009207526' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:54:08.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:08 smithi114 ceph-mon[25759]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-12T21:54:08.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:08 smithi114 ceph-mon[25759]: Cluster is now healthy 2024-04-12T21:54:08.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:08 smithi097 ceph-mon[24835]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-12T21:54:08.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:08 smithi097 ceph-mon[24835]: Cluster is now healthy 2024-04-12T21:54:08.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:08 smithi097 ceph-mon[29620]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-12T21:54:08.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:08 smithi097 ceph-mon[29620]: Cluster is now healthy 2024-04-12T21:54:09.607 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:09 smithi114 ceph-mon[25759]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-04-12T21:54:09.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:09 smithi097 ceph-mon[24835]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-04-12T21:54:09.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:09 smithi097 ceph-mon[29620]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-04-12T21:54:11.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:11 smithi097 ceph-mon[24835]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-04-12T21:54:11.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:11 smithi097 ceph-mon[29620]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-04-12T21:54:11.695 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:11 smithi114 ceph-mon[25759]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-04-12T21:54:12.431 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:12 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-12T21:54:12.431 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:12 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:54:12.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:12 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-12T21:54:12.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:12 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:54:12.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:12 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-12T21:54:12.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:12 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:54:13.614 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:13 smithi114 ceph-mon[25759]: Deploying daemon osd.6 on smithi114 2024-04-12T21:54:13.615 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:13 smithi114 ceph-mon[25759]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-12T21:54:13.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:13 smithi097 ceph-mon[24835]: Deploying daemon osd.6 on smithi114 2024-04-12T21:54:13.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:13 smithi097 ceph-mon[24835]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-12T21:54:13.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:13 smithi097 ceph-mon[29620]: Deploying daemon osd.6 on smithi114 2024-04-12T21:54:13.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:13 smithi097 ceph-mon[29620]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-12T21:54:15.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:15 smithi097 ceph-mon[24835]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-04-12T21:54:15.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:15 smithi097 ceph-mon[29620]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-04-12T21:54:15.672 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:15 smithi114 ceph-mon[25759]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-04-12T21:54:17.639 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:17 smithi114 ceph-mon[25759]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-04-12T21:54:17.640 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:17 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:17.640 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:17 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:54:17.640 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:17 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:54:17.640 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:17 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:54:17.669 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:17 smithi097 ceph-mon[24835]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-04-12T21:54:17.669 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:17 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:17.669 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:17 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:54:17.669 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:17 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:54:17.669 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:17 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:54:17.669 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:17 smithi097 ceph-mon[29620]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-04-12T21:54:17.669 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:17 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:17.670 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:17 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:54:17.670 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:17 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:54:17.670 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:17 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:54:19.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:19 smithi114 ceph-mon[25759]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-04-12T21:54:19.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:19 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:19.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:19 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:19.529 INFO:teuthology.orchestra.run.smithi114.stdout:Created osd(s) 6 on host 'smithi114' 2024-04-12T21:54:19.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:19 smithi097 ceph-mon[24835]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-04-12T21:54:19.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:19 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:19.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:19 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:19.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:19 smithi097 ceph-mon[29620]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-04-12T21:54:19.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:19 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:19.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:19 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:20.127 DEBUG:teuthology.orchestra.run.smithi114:osd.6> sudo journalctl -f -n 0 -u ceph-80bf7680-f916-11ee-bc8d-c7b262605968@osd.6.service 2024-04-12T21:54:20.129 INFO:tasks.cephadm:Deploying osd.7 on smithi114 with /dev/vg_nvme/lv_1... 2024-04-12T21:54:20.129 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-12T21:54:20.669 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:20 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:20.669 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:20 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:54:20.669 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:20 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:54:20.669 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:20 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:54:20.669 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:20 smithi114 ceph-mon[25759]: from='osd.6 [v2:172.21.15.114:6816/2512498411,v1:172.21.15.114:6817/2512498411]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-12T21:54:20.669 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:20 smithi114 ceph-mon[25759]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-12T21:54:20.669 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:20 smithi114 ceph-mon[25759]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-12T21:54:20.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:20 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:20.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:20 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:54:20.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:20 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:54:20.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:20 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:54:20.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:20 smithi097 ceph-mon[24835]: from='osd.6 [v2:172.21.15.114:6816/2512498411,v1:172.21.15.114:6817/2512498411]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-12T21:54:20.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:20 smithi097 ceph-mon[24835]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-12T21:54:20.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:20 smithi097 ceph-mon[24835]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-12T21:54:20.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:20 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:20.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:20 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:54:20.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:20 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:54:20.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:20 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:54:20.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:20 smithi097 ceph-mon[29620]: from='osd.6 [v2:172.21.15.114:6816/2512498411,v1:172.21.15.114:6817/2512498411]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-12T21:54:20.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:20 smithi097 ceph-mon[29620]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-12T21:54:20.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:20 smithi097 ceph-mon[29620]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-12T21:54:21.901 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:21 smithi114 ceph-mon[25759]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-12T21:54:21.901 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:21 smithi114 ceph-mon[25759]: osdmap e44: 7 total, 6 up, 7 in 2024-04-12T21:54:21.901 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:21 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-12T21:54:21.901 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:21 smithi114 ceph-mon[25759]: from='osd.6 [v2:172.21.15.114:6816/2512498411,v1:172.21.15.114:6817/2512498411]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-12T21:54:21.901 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:21 smithi114 ceph-mon[25759]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-12T21:54:21.902 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 21:54:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[34949]: 2024-04-12T21:54:21.541+0000 7ff54f56e700 -1 osd.6 0 waiting for initial osdmap 2024-04-12T21:54:21.902 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 21:54:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[34949]: 2024-04-12T21:54:21.548+0000 7ff54a706700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T21:54:21.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:21 smithi097 ceph-mon[29620]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-12T21:54:21.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:21 smithi097 ceph-mon[29620]: osdmap e44: 7 total, 6 up, 7 in 2024-04-12T21:54:21.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:21 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-12T21:54:21.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:21 smithi097 ceph-mon[29620]: from='osd.6 [v2:172.21.15.114:6816/2512498411,v1:172.21.15.114:6817/2512498411]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-12T21:54:21.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:21 smithi097 ceph-mon[29620]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-12T21:54:21.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:21 smithi097 ceph-mon[24835]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-12T21:54:21.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:21 smithi097 ceph-mon[24835]: osdmap e44: 7 total, 6 up, 7 in 2024-04-12T21:54:21.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:21 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-12T21:54:21.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:21 smithi097 ceph-mon[24835]: from='osd.6 [v2:172.21.15.114:6816/2512498411,v1:172.21.15.114:6817/2512498411]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-12T21:54:21.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:21 smithi097 ceph-mon[24835]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-12T21:54:22.707 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:22 smithi114 ceph-mon[25759]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-04-12T21:54:22.707 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:22 smithi114 ceph-mon[25759]: osdmap e45: 7 total, 6 up, 7 in 2024-04-12T21:54:22.707 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:22 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-12T21:54:22.707 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:22 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-12T21:54:22.707 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:22 smithi114 ceph-mon[25759]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-12T21:54:22.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:22 smithi097 ceph-mon[24835]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-04-12T21:54:22.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:22 smithi097 ceph-mon[24835]: osdmap e45: 7 total, 6 up, 7 in 2024-04-12T21:54:22.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:22 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-12T21:54:22.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:22 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-12T21:54:22.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:22 smithi097 ceph-mon[24835]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-12T21:54:22.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:22 smithi097 ceph-mon[29620]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-04-12T21:54:22.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:22 smithi097 ceph-mon[29620]: osdmap e45: 7 total, 6 up, 7 in 2024-04-12T21:54:22.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:22 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-12T21:54:22.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:22 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-12T21:54:22.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:22 smithi097 ceph-mon[29620]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-12T21:54:23.186 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-12T21:54:23.186 INFO:teuthology.orchestra.run.smithi114.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-12T21:54:23.186 INFO:teuthology.orchestra.run.smithi114.stderr: stderr: 10+0 records in 2024-04-12T21:54:23.187 INFO:teuthology.orchestra.run.smithi114.stderr:10+0 records out 2024-04-12T21:54:23.187 INFO:teuthology.orchestra.run.smithi114.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0150624 s, 696 MB/s 2024-04-12T21:54:23.187 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping successful for: 2024-04-12T21:54:23.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:23 smithi097 ceph-mon[24835]: purged_snaps scrub starts 2024-04-12T21:54:23.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:23 smithi097 ceph-mon[24835]: purged_snaps scrub ok 2024-04-12T21:54:23.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:23 smithi097 ceph-mon[24835]: osd.6 [v2:172.21.15.114:6816/2512498411,v1:172.21.15.114:6817/2512498411] boot 2024-04-12T21:54:23.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:23 smithi097 ceph-mon[24835]: osdmap e46: 7 total, 7 up, 7 in 2024-04-12T21:54:23.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:23 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-12T21:54:23.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:23 smithi097 ceph-mon[24835]: osdmap e47: 7 total, 7 up, 7 in 2024-04-12T21:54:23.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:23 smithi097 ceph-mon[29620]: purged_snaps scrub starts 2024-04-12T21:54:23.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:23 smithi097 ceph-mon[29620]: purged_snaps scrub ok 2024-04-12T21:54:23.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:23 smithi097 ceph-mon[29620]: osd.6 [v2:172.21.15.114:6816/2512498411,v1:172.21.15.114:6817/2512498411] boot 2024-04-12T21:54:23.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:23 smithi097 ceph-mon[29620]: osdmap e46: 7 total, 7 up, 7 in 2024-04-12T21:54:23.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:23 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-12T21:54:23.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:23 smithi097 ceph-mon[29620]: osdmap e47: 7 total, 7 up, 7 in 2024-04-12T21:54:23.933 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:23 smithi114 ceph-mon[25759]: purged_snaps scrub starts 2024-04-12T21:54:23.933 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:23 smithi114 ceph-mon[25759]: purged_snaps scrub ok 2024-04-12T21:54:23.933 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:23 smithi114 ceph-mon[25759]: osd.6 [v2:172.21.15.114:6816/2512498411,v1:172.21.15.114:6817/2512498411] boot 2024-04-12T21:54:23.933 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:23 smithi114 ceph-mon[25759]: osdmap e46: 7 total, 7 up, 7 in 2024-04-12T21:54:23.933 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:23 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-12T21:54:23.934 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:23 smithi114 ceph-mon[25759]: osdmap e47: 7 total, 7 up, 7 in 2024-04-12T21:54:24.028 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- ceph orch daemon add osd smithi114:vg_nvme/lv_1 2024-04-12T21:54:25.583 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:25 smithi114 ceph-mon[25759]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-12T21:54:25.584 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:25 smithi114 ceph-mon[25759]: osdmap e48: 7 total, 7 up, 7 in 2024-04-12T21:54:25.584 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:25 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:25.584 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:25 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:54:25.584 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:25 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:54:25.584 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:25 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:54:25.584 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:25 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:25.584 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:25 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:25.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:25 smithi097 ceph-mon[24835]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-12T21:54:25.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:25 smithi097 ceph-mon[24835]: osdmap e48: 7 total, 7 up, 7 in 2024-04-12T21:54:25.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:25 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:25.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:25 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:54:25.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:25 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:54:25.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:25 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:54:25.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:25 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:25.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:25 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:25.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:25 smithi097 ceph-mon[29620]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-12T21:54:25.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:25 smithi097 ceph-mon[29620]: osdmap e48: 7 total, 7 up, 7 in 2024-04-12T21:54:25.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:25 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:25.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:25 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:54:25.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:25 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:54:25.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:25 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:54:25.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:25 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:25.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:25 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:26.624 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:26 smithi114 ceph-mon[25759]: Detected new or changed devices on smithi114 2024-04-12T21:54:26.624 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:26 smithi114 ceph-mon[25759]: Adjusting osd_memory_target on smithi114 to 5696M 2024-04-12T21:54:26.624 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:26 smithi114 ceph-mon[25759]: osdmap e49: 7 total, 7 up, 7 in 2024-04-12T21:54:26.624 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:26 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:54:26.624 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:26 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:54:26.624 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:26 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:54:26.624 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:26 smithi114 ceph-mon[25759]: osdmap e50: 7 total, 7 up, 7 in 2024-04-12T21:54:26.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:26 smithi097 ceph-mon[24835]: Detected new or changed devices on smithi114 2024-04-12T21:54:26.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:26 smithi097 ceph-mon[24835]: Adjusting osd_memory_target on smithi114 to 5696M 2024-04-12T21:54:26.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:26 smithi097 ceph-mon[24835]: osdmap e49: 7 total, 7 up, 7 in 2024-04-12T21:54:26.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:26 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:54:26.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:26 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:54:26.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:26 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:54:26.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:26 smithi097 ceph-mon[24835]: osdmap e50: 7 total, 7 up, 7 in 2024-04-12T21:54:26.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:26 smithi097 ceph-mon[29620]: Detected new or changed devices on smithi114 2024-04-12T21:54:26.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:26 smithi097 ceph-mon[29620]: Adjusting osd_memory_target on smithi114 to 5696M 2024-04-12T21:54:26.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:26 smithi097 ceph-mon[29620]: osdmap e49: 7 total, 7 up, 7 in 2024-04-12T21:54:26.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:26 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:54:26.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:26 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:54:26.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:26 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:54:26.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:26 smithi097 ceph-mon[29620]: osdmap e50: 7 total, 7 up, 7 in 2024-04-12T21:54:27.425 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:27 smithi114 ceph-mon[25759]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-12T21:54:27.426 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:27 smithi114 ceph-mon[25759]: from='client.24269 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:54:27.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:27 smithi097 ceph-mon[24835]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-12T21:54:27.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:27 smithi097 ceph-mon[24835]: from='client.24269 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:54:27.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:27 smithi097 ceph-mon[29620]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-12T21:54:27.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:27 smithi097 ceph-mon[29620]: from='client.24269 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:54:29.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:29 smithi097 ceph-mon[24835]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-12T21:54:29.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:29 smithi097 ceph-mon[24835]: from='client.? 172.21.15.114:0/3498201153' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "719dfe0e-c27b-4ada-a3e0-e381d70eb53b"}]: dispatch 2024-04-12T21:54:29.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:29 smithi097 ceph-mon[24835]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "719dfe0e-c27b-4ada-a3e0-e381d70eb53b"}]: dispatch 2024-04-12T21:54:29.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:29 smithi097 ceph-mon[24835]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "719dfe0e-c27b-4ada-a3e0-e381d70eb53b"}]': finished 2024-04-12T21:54:29.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:29 smithi097 ceph-mon[24835]: osdmap e51: 8 total, 7 up, 8 in 2024-04-12T21:54:29.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:29 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-12T21:54:29.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:29 smithi097 ceph-mon[24835]: from='client.? 172.21.15.114:0/2888829610' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:54:29.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:29 smithi097 ceph-mon[29620]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-12T21:54:29.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:29 smithi097 ceph-mon[29620]: from='client.? 172.21.15.114:0/3498201153' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "719dfe0e-c27b-4ada-a3e0-e381d70eb53b"}]: dispatch 2024-04-12T21:54:29.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:29 smithi097 ceph-mon[29620]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "719dfe0e-c27b-4ada-a3e0-e381d70eb53b"}]: dispatch 2024-04-12T21:54:29.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:29 smithi097 ceph-mon[29620]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "719dfe0e-c27b-4ada-a3e0-e381d70eb53b"}]': finished 2024-04-12T21:54:29.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:29 smithi097 ceph-mon[29620]: osdmap e51: 8 total, 7 up, 8 in 2024-04-12T21:54:29.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:29 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-12T21:54:29.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:29 smithi097 ceph-mon[29620]: from='client.? 172.21.15.114:0/2888829610' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:54:29.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:29 smithi114 ceph-mon[25759]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-12T21:54:29.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:29 smithi114 ceph-mon[25759]: from='client.? 172.21.15.114:0/3498201153' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "719dfe0e-c27b-4ada-a3e0-e381d70eb53b"}]: dispatch 2024-04-12T21:54:29.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:29 smithi114 ceph-mon[25759]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "719dfe0e-c27b-4ada-a3e0-e381d70eb53b"}]: dispatch 2024-04-12T21:54:29.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:29 smithi114 ceph-mon[25759]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "719dfe0e-c27b-4ada-a3e0-e381d70eb53b"}]': finished 2024-04-12T21:54:29.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:29 smithi114 ceph-mon[25759]: osdmap e51: 8 total, 7 up, 8 in 2024-04-12T21:54:29.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:29 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-12T21:54:29.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:29 smithi114 ceph-mon[25759]: from='client.? 172.21.15.114:0/2888829610' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:54:31.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:31 smithi114 ceph-mon[25759]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-04-12T21:54:31.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:31 smithi097 ceph-mon[24835]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-04-12T21:54:31.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:31 smithi097 ceph-mon[29620]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-04-12T21:54:33.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:33 smithi097 ceph-mon[24835]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-04-12T21:54:33.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:33 smithi097 ceph-mon[29620]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-04-12T21:54:33.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:33 smithi114 ceph-mon[25759]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-04-12T21:54:35.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:35 smithi097 ceph-mon[24835]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-12T21:54:35.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:35 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-12T21:54:35.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:35 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:54:35.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:35 smithi097 ceph-mon[29620]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-12T21:54:35.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:35 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-12T21:54:35.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:35 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:54:35.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:35 smithi114 ceph-mon[25759]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-12T21:54:35.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:35 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-12T21:54:35.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:35 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:54:36.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:36 smithi114 ceph-mon[25759]: Deploying daemon osd.7 on smithi114 2024-04-12T21:54:36.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:36 smithi097 ceph-mon[24835]: Deploying daemon osd.7 on smithi114 2024-04-12T21:54:36.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:36 smithi097 ceph-mon[29620]: Deploying daemon osd.7 on smithi114 2024-04-12T21:54:37.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:37 smithi097 ceph-mon[24835]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-04-12T21:54:37.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:37 smithi097 ceph-mon[29620]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-04-12T21:54:37.675 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:37 smithi114 ceph-mon[25759]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-04-12T21:54:39.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:39 smithi097 ceph-mon[24835]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-12T21:54:39.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:39 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:39.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:39 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:54:39.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:39 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:54:39.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:39 smithi097 ceph-mon[29620]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-12T21:54:39.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:39 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:39.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:39 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:54:39.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:39 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:54:39.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:39 smithi114 ceph-mon[25759]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-12T21:54:39.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:39 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:39.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:39 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:54:39.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:39 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:54:40.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:40 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:54:40.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:40 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:54:40.729 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:40 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:54:41.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:41 smithi097 ceph-mon[24835]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 44 KiB/s, 0 objects/s recovering 2024-04-12T21:54:41.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:41 smithi097 ceph-mon[29620]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 44 KiB/s, 0 objects/s recovering 2024-04-12T21:54:41.719 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:41 smithi114 ceph-mon[25759]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 44 KiB/s, 0 objects/s recovering 2024-04-12T21:54:42.728 INFO:teuthology.orchestra.run.smithi114.stdout:Created osd(s) 7 on host 'smithi114' 2024-04-12T21:54:42.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:42 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:42.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:42 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:42.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:42 smithi114 ceph-mon[25759]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-12T21:54:42.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:42 smithi114 ceph-mon[25759]: from='osd.7 [v2:172.21.15.114:6824/1147837664,v1:172.21.15.114:6825/1147837664]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-12T21:54:42.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:42 smithi114 ceph-mon[25759]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-12T21:54:42.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:42 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:42.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:42 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:54:42.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:42 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:54:42.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:42 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:54:43.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:42 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:43.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:42 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:43.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:42 smithi097 ceph-mon[24835]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-12T21:54:43.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:42 smithi097 ceph-mon[24835]: from='osd.7 [v2:172.21.15.114:6824/1147837664,v1:172.21.15.114:6825/1147837664]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-12T21:54:43.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:42 smithi097 ceph-mon[24835]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-12T21:54:43.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:42 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:43.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:42 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:54:43.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:42 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:54:43.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:42 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:54:43.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:42 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:43.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:42 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:43.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:42 smithi097 ceph-mon[29620]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-12T21:54:43.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:42 smithi097 ceph-mon[29620]: from='osd.7 [v2:172.21.15.114:6824/1147837664,v1:172.21.15.114:6825/1147837664]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-12T21:54:43.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:42 smithi097 ceph-mon[29620]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-12T21:54:43.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:42 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:43.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:42 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:54:43.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:42 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:54:43.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:42 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:54:43.378 DEBUG:teuthology.orchestra.run.smithi114:osd.7> sudo journalctl -f -n 0 -u ceph-80bf7680-f916-11ee-bc8d-c7b262605968@osd.7.service 2024-04-12T21:54:43.383 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-04-12T21:54:43.383 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- ceph osd stat -f json 2024-04-12T21:54:43.950 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:43 smithi097 ceph-mon[29620]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-12T21:54:43.950 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:43 smithi097 ceph-mon[29620]: osdmap e52: 8 total, 7 up, 8 in 2024-04-12T21:54:43.950 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:43 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-12T21:54:43.951 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:43 smithi097 ceph-mon[29620]: from='osd.7 [v2:172.21.15.114:6824/1147837664,v1:172.21.15.114:6825/1147837664]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-12T21:54:43.951 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:43 smithi097 ceph-mon[29620]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-12T21:54:43.951 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:43 smithi097 ceph-mon[24835]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-12T21:54:43.951 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:43 smithi097 ceph-mon[24835]: osdmap e52: 8 total, 7 up, 8 in 2024-04-12T21:54:43.951 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:43 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-12T21:54:43.951 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:43 smithi097 ceph-mon[24835]: from='osd.7 [v2:172.21.15.114:6824/1147837664,v1:172.21.15.114:6825/1147837664]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-12T21:54:43.951 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:43 smithi097 ceph-mon[24835]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-12T21:54:43.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:43 smithi114 ceph-mon[25759]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-12T21:54:43.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:43 smithi114 ceph-mon[25759]: osdmap e52: 8 total, 7 up, 8 in 2024-04-12T21:54:43.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:43 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-12T21:54:43.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:43 smithi114 ceph-mon[25759]: from='osd.7 [v2:172.21.15.114:6824/1147837664,v1:172.21.15.114:6825/1147837664]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-12T21:54:43.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:43 smithi114 ceph-mon[25759]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-12T21:54:43.995 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 21:54:43 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[37937]: 2024-04-12T21:54:43.914+0000 7f1f8fbd0700 -1 osd.7 0 waiting for initial osdmap 2024-04-12T21:54:43.995 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 21:54:43 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[37937]: 2024-04-12T21:54:43.921+0000 7f1f88563700 -1 osd.7 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T21:54:45.065 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:44 smithi097 ceph-mon[24835]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-04-12T21:54:45.066 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:44 smithi097 ceph-mon[24835]: osdmap e53: 8 total, 7 up, 8 in 2024-04-12T21:54:45.066 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:44 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-12T21:54:45.066 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:44 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-12T21:54:45.066 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:44 smithi097 ceph-mon[24835]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-12T21:54:45.066 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:44 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T21:54:45.066 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:44 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T21:54:45.066 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:44 smithi097 ceph-mon[29620]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-04-12T21:54:45.066 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:44 smithi097 ceph-mon[29620]: osdmap e53: 8 total, 7 up, 8 in 2024-04-12T21:54:45.066 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:44 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-12T21:54:45.067 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:44 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-12T21:54:45.067 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:44 smithi097 ceph-mon[29620]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-12T21:54:45.067 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:44 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T21:54:45.067 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:44 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T21:54:45.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:44 smithi114 ceph-mon[25759]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-04-12T21:54:45.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:44 smithi114 ceph-mon[25759]: osdmap e53: 8 total, 7 up, 8 in 2024-04-12T21:54:45.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:44 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-12T21:54:45.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:44 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-12T21:54:45.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:44 smithi114 ceph-mon[25759]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-12T21:54:45.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:44 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T21:54:45.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:44 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T21:54:45.465 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-12T21:54:46.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:45 smithi097 ceph-mon[24835]: purged_snaps scrub starts 2024-04-12T21:54:46.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:45 smithi097 ceph-mon[24835]: purged_snaps scrub ok 2024-04-12T21:54:46.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:45 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-12T21:54:46.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:45 smithi097 ceph-mon[24835]: osd.7 [v2:172.21.15.114:6824/1147837664,v1:172.21.15.114:6825/1147837664] boot 2024-04-12T21:54:46.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:45 smithi097 ceph-mon[24835]: osdmap e54: 8 total, 8 up, 8 in 2024-04-12T21:54:46.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:45 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-12T21:54:46.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:45 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/1276742310' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-12T21:54:46.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:45 smithi097 ceph-mon[29620]: purged_snaps scrub starts 2024-04-12T21:54:46.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:45 smithi097 ceph-mon[29620]: purged_snaps scrub ok 2024-04-12T21:54:46.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:45 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-12T21:54:46.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:45 smithi097 ceph-mon[29620]: osd.7 [v2:172.21.15.114:6824/1147837664,v1:172.21.15.114:6825/1147837664] boot 2024-04-12T21:54:46.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:45 smithi097 ceph-mon[29620]: osdmap e54: 8 total, 8 up, 8 in 2024-04-12T21:54:46.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:45 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-12T21:54:46.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:45 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/1276742310' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-12T21:54:46.174 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:45 smithi114 ceph-mon[25759]: purged_snaps scrub starts 2024-04-12T21:54:46.174 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:45 smithi114 ceph-mon[25759]: purged_snaps scrub ok 2024-04-12T21:54:46.174 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:45 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-12T21:54:46.174 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:45 smithi114 ceph-mon[25759]: osd.7 [v2:172.21.15.114:6824/1147837664,v1:172.21.15.114:6825/1147837664] boot 2024-04-12T21:54:46.174 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:45 smithi114 ceph-mon[25759]: osdmap e54: 8 total, 8 up, 8 in 2024-04-12T21:54:46.174 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:45 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-12T21:54:46.174 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:45 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/1276742310' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-12T21:54:46.234 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":54,"num_osds":8,"num_up_osds":8,"osd_up_since":1712958884,"num_in_osds":8,"osd_in_since":1712958868,"num_remapped_pgs":1} 2024-04-12T21:54:46.234 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- ceph osd dump --format=json 2024-04-12T21:54:46.452 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:54:47.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:46 smithi097 ceph-mon[24835]: osdmap e55: 8 total, 8 up, 8 in 2024-04-12T21:54:47.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:46 smithi097 ceph-mon[24835]: pgmap v166: 1 pgs: 1 remapped+peering; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:54:47.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:46 smithi097 ceph-mon[24835]: Detected new or changed devices on smithi114 2024-04-12T21:54:47.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:46 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:47.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:46 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:54:47.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:46 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:54:47.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:46 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:54:47.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:46 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:54:47.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:46 smithi097 ceph-mon[24835]: Adjusting osd_memory_target on smithi114 to 4272M 2024-04-12T21:54:47.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:46 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:47.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:46 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:47.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:46 smithi097 ceph-mon[29620]: osdmap e55: 8 total, 8 up, 8 in 2024-04-12T21:54:47.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:46 smithi097 ceph-mon[29620]: pgmap v166: 1 pgs: 1 remapped+peering; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:54:47.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:46 smithi097 ceph-mon[29620]: Detected new or changed devices on smithi114 2024-04-12T21:54:47.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:46 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:47.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:46 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:54:47.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:46 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:54:47.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:46 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:54:47.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:46 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:54:47.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:46 smithi097 ceph-mon[29620]: Adjusting osd_memory_target on smithi114 to 4272M 2024-04-12T21:54:47.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:46 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:47.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:46 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:47.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:46 smithi114 ceph-mon[25759]: osdmap e55: 8 total, 8 up, 8 in 2024-04-12T21:54:47.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:46 smithi114 ceph-mon[25759]: pgmap v166: 1 pgs: 1 remapped+peering; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:54:47.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:46 smithi114 ceph-mon[25759]: Detected new or changed devices on smithi114 2024-04-12T21:54:47.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:46 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:47.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:46 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:54:47.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:46 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:54:47.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:46 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:54:47.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:46 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:54:47.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:46 smithi114 ceph-mon[25759]: Adjusting osd_memory_target on smithi114 to 4272M 2024-04-12T21:54:47.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:46 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:47.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:46 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:48.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:47 smithi114 ceph-mon[25759]: osdmap e56: 8 total, 8 up, 8 in 2024-04-12T21:54:48.336 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-12T21:54:48.336 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:47 smithi097 ceph-mon[24835]: osdmap e56: 8 total, 8 up, 8 in 2024-04-12T21:54:48.337 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:47 smithi097 ceph-mon[29620]: osdmap e56: 8 total, 8 up, 8 in 2024-04-12T21:54:48.337 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":57,"fsid":"80bf7680-f916-11ee-bc8d-c7b262605968","created":"2024-04-12T21:49:48.426985+0000","modified":"2024-04-12T21:54:47.925583+0000","last_up_change":"2024-04-12T21:54:44.911811+0000","last_in_change":"2024-04-12T21:54:28.414803+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-12T21:52:54.377978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"315f320f-95fd-4b18-ab6a-417ba7651667","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":2413859098},{"type":"v1","addr":"172.21.15.97:6803","nonce":2413859098}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":2413859098},{"type":"v1","addr":"172.21.15.97:6805","nonce":2413859098}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2413859098},{"type":"v1","addr":"172.21.15.97:6809","nonce":2413859098}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":2413859098},{"type":"v1","addr":"172.21.15.97:6807","nonce":2413859098}]},"public_addr":"172.21.15.97:6803/2413859098","cluster_addr":"172.21.15.97:6805/2413859098","heartbeat_back_addr":"172.21.15.97:6809/2413859098","heartbeat_front_addr":"172.21.15.97:6807/2413859098","state":["exists","up"]},{"osd":1,"uuid":"c58dc94b-86d4-4b83-95ad-d3779a5aa46e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":1786985012},{"type":"v1","addr":"172.21.15.97:6811","nonce":1786985012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":1786985012},{"type":"v1","addr":"172.21.15.97:6813","nonce":1786985012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":1786985012},{"type":"v1","addr":"172.21.15.97:6817","nonce":1786985012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":1786985012},{"type":"v1","addr":"172.21.15.97:6815","nonce":1786985012}]},"public_addr":"172.21.15.97:6811/1786985012","cluster_addr":"172.21.15.97:6813/1786985012","heartbeat_back_addr":"172.21.15.97:6817/1786985012","heartbeat_front_addr":"172.21.15.97:6815/1786985012","state":["exists","up"]},{"osd":2,"uuid":"4920ee2a-de4d-4072-92bb-dae616a76152","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":1944472297},{"type":"v1","addr":"172.21.15.97:6819","nonce":1944472297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":1944472297},{"type":"v1","addr":"172.21.15.97:6821","nonce":1944472297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1944472297},{"type":"v1","addr":"172.21.15.97:6825","nonce":1944472297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":1944472297},{"type":"v1","addr":"172.21.15.97:6823","nonce":1944472297}]},"public_addr":"172.21.15.97:6819/1944472297","cluster_addr":"172.21.15.97:6821/1944472297","heartbeat_back_addr":"172.21.15.97:6825/1944472297","heartbeat_front_addr":"172.21.15.97:6823/1944472297","state":["exists","up"]},{"osd":3,"uuid":"928b3bd7-c0df-443c-9a28-5ceb62ae3160","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":75554646},{"type":"v1","addr":"172.21.15.97:6827","nonce":75554646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":75554646},{"type":"v1","addr":"172.21.15.97:6829","nonce":75554646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":75554646},{"type":"v1","addr":"172.21.15.97:6833","nonce":75554646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":75554646},{"type":"v1","addr":"172.21.15.97:6831","nonce":75554646}]},"public_addr":"172.21.15.97:6827/75554646","cluster_addr":"172.21.15.97:6829/75554646","heartbeat_back_addr":"172.21.15.97:6833/75554646","heartbeat_front_addr":"172.21.15.97:6831/75554646","state":["exists","up"]},{"osd":4,"uuid":"02f49773-bcfc-499c-9174-d9cb5991c789","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":2190104221},{"type":"v1","addr":"172.21.15.114:6801","nonce":2190104221}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":2190104221},{"type":"v1","addr":"172.21.15.114:6803","nonce":2190104221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":2190104221},{"type":"v1","addr":"172.21.15.114:6807","nonce":2190104221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":2190104221},{"type":"v1","addr":"172.21.15.114:6805","nonce":2190104221}]},"public_addr":"172.21.15.114:6801/2190104221","cluster_addr":"172.21.15.114:6803/2190104221","heartbeat_back_addr":"172.21.15.114:6807/2190104221","heartbeat_front_addr":"172.21.15.114:6805/2190104221","state":["exists","up"]},{"osd":5,"uuid":"997405e6-e6f2-4c40-b81c-7848c56c062c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":515183706},{"type":"v1","addr":"172.21.15.114:6809","nonce":515183706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":515183706},{"type":"v1","addr":"172.21.15.114:6811","nonce":515183706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":515183706},{"type":"v1","addr":"172.21.15.114:6815","nonce":515183706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":515183706},{"type":"v1","addr":"172.21.15.114:6813","nonce":515183706}]},"public_addr":"172.21.15.114:6809/515183706","cluster_addr":"172.21.15.114:6811/515183706","heartbeat_back_addr":"172.21.15.114:6815/515183706","heartbeat_front_addr":"172.21.15.114:6813/515183706","state":["exists","up"]},{"osd":6,"uuid":"68b36dad-0aff-40a4-b8f7-8c1dcf42a5c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":2512498411},{"type":"v1","addr":"172.21.15.114:6817","nonce":2512498411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":2512498411},{"type":"v1","addr":"172.21.15.114:6819","nonce":2512498411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":2512498411},{"type":"v1","addr":"172.21.15.114:6823","nonce":2512498411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":2512498411},{"type":"v1","addr":"172.21.15.114:6821","nonce":2512498411}]},"public_addr":"172.21.15.114:6817/2512498411","cluster_addr":"172.21.15.114:6819/2512498411","heartbeat_back_addr":"172.21.15.114:6823/2512498411","heartbeat_front_addr":"172.21.15.114:6821/2512498411","state":["exists","up"]},{"osd":7,"uuid":"719dfe0e-c27b-4ada-a3e0-e381d70eb53b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":1147837664},{"type":"v1","addr":"172.21.15.114:6825","nonce":1147837664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":1147837664},{"type":"v1","addr":"172.21.15.114:6827","nonce":1147837664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":1147837664},{"type":"v1","addr":"172.21.15.114:6831","nonce":1147837664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":1147837664},{"type":"v1","addr":"172.21.15.114:6829","nonce":1147837664}]},"public_addr":"172.21.15.114:6825/1147837664","cluster_addr":"172.21.15.114:6827/1147837664","heartbeat_back_addr":"172.21.15.114:6831/1147837664","heartbeat_front_addr":"172.21.15.114:6829/1147837664","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:52:02.934510+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:52:26.942832+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:52:51.162399+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:53:15.754757+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:53:36.448575+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:53:58.560201+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:54:20.557106+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:54:43.500361+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.97:0/831596468":"2024-04-13T21:50:43.236028+0000","172.21.15.97:0/1562080679":"2024-04-13T21:50:43.236028+0000","172.21.15.97:6801/258588252":"2024-04-13T21:50:43.236028+0000","172.21.15.97:0/2696421935":"2024-04-13T21:50:43.236028+0000","172.21.15.97:6800/258588252":"2024-04-13T21:50:43.236028+0000","172.21.15.97:0/2460245404":"2024-04-13T21:50:20.870746+0000","172.21.15.97:0/1828347223":"2024-04-13T21:50:20.870746+0000","172.21.15.97:6800/1796168157":"2024-04-13T21:50:20.870746+0000","172.21.15.97:6801/1796168157":"2024-04-13T21:50:20.870746+0000","172.21.15.97:0/1917553394":"2024-04-13T21:50:20.870746+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-12T21:54:48.992 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-12T21:52:54.377978+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-04-12T21:54:48.992 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-12T21:54:49.205 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:54:49.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:48 smithi114 ceph-mon[25759]: osdmap e57: 8 total, 8 up, 8 in 2024-04-12T21:54:49.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:48 smithi114 ceph-mon[25759]: pgmap v169: 1 pgs: 1 remapped+peering; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:54:49.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:48 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/2577357275' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:54:49.328 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:48 smithi097 ceph-mon[24835]: osdmap e57: 8 total, 8 up, 8 in 2024-04-12T21:54:49.328 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:48 smithi097 ceph-mon[24835]: pgmap v169: 1 pgs: 1 remapped+peering; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:54:49.328 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:48 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/2577357275' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:54:49.328 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:48 smithi097 ceph-mon[29620]: osdmap e57: 8 total, 8 up, 8 in 2024-04-12T21:54:49.329 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:48 smithi097 ceph-mon[29620]: pgmap v169: 1 pgs: 1 remapped+peering; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:54:49.329 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:48 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/2577357275' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:54:50.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:49 smithi114 ceph-mon[25759]: osdmap e58: 8 total, 8 up, 8 in 2024-04-12T21:54:50.398 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:49 smithi097 ceph-mon[24835]: osdmap e58: 8 total, 8 up, 8 in 2024-04-12T21:54:50.399 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:49 smithi097 ceph-mon[29620]: osdmap e58: 8 total, 8 up, 8 in 2024-04-12T21:54:51.045 INFO:teuthology.orchestra.run.smithi097.stdout:pg_num: 1 2024-04-12T21:54:51.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:50 smithi114 ceph-mon[25759]: pgmap v171: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:54:51.355 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:50 smithi097 ceph-mon[24835]: pgmap v171: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:54:51.355 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:50 smithi097 ceph-mon[29620]: pgmap v171: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:54:51.783 INFO:tasks.cephadm:Adding prometheus.a on smithi114 2024-04-12T21:54:51.783 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- ceph orch apply prometheus '1;smithi114=a' 2024-04-12T21:54:52.100 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:51 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/2512180548' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-12T21:54:52.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:51 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/2512180548' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-12T21:54:52.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:51 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/2512180548' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-12T21:54:53.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:52 smithi114 ceph-mon[25759]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-12T21:54:53.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:52 smithi097 ceph-mon[24835]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-12T21:54:53.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:52 smithi097 ceph-mon[29620]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-12T21:54:53.683 INFO:teuthology.orchestra.run.smithi114.stdout:Scheduled prometheus update... 2024-04-12T21:54:54.286 DEBUG:teuthology.orchestra.run.smithi114:prometheus.a> sudo journalctl -f -n 0 -u ceph-80bf7680-f916-11ee-bc8d-c7b262605968@prometheus.a.service 2024-04-12T21:54:54.288 INFO:tasks.cephadm:Adding node-exporter.a on smithi097 2024-04-12T21:54:54.288 INFO:tasks.cephadm:Adding node-exporter.b on smithi114 2024-04-12T21:54:54.288 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- ceph orch apply node-exporter '2;smithi097=a;smithi114=b' 2024-04-12T21:54:54.882 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:54:54 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: ignoring --setuser ceph since I am not root 2024-04-12T21:54:54.883 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:54:54 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: ignoring --setgroup ceph since I am not root 2024-04-12T21:54:54.883 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:54:54 smithi114 ceph-mgr[26943]: -- 172.21.15.114:0/1192662016 <== mon.2 v2:172.21.15.114:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559a47846340 con 0x559a4781ec00 2024-04-12T21:54:54.883 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:54 smithi114 ceph-mon[25759]: from='client.24308 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi114=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:54:54.883 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:54 smithi114 ceph-mon[25759]: Saving service prometheus spec with placement smithi114=a;count:1 2024-04-12T21:54:54.883 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:54 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:54.883 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:54 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:54:54.883 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:54 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:54:54.883 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:54 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:54:54.883 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:54 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:54.884 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:54 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-12T21:54:54.884 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:54 smithi114 ceph-mon[25759]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-12T21:54:55.161 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:54:54 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: ignoring --setuser ceph since I am not root 2024-04-12T21:54:55.161 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:54:54 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: ignoring --setgroup ceph since I am not root 2024-04-12T21:54:55.162 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:54:54 smithi097 ceph-mgr[25072]: -- 172.21.15.97:0/3701229445 <== mon.0 v2:172.21.15.97:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f322dc0340 con 0x55f322d98800 2024-04-12T21:54:55.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:54 smithi097 ceph-mon[29620]: from='client.24308 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi114=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:54:55.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:54 smithi097 ceph-mon[29620]: Saving service prometheus spec with placement smithi114=a;count:1 2024-04-12T21:54:55.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:54 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:55.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:54 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:54:55.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:54 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:54:55.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:54 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:54:55.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:54 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:55.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:54 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-12T21:54:55.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:54 smithi097 ceph-mon[29620]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-12T21:54:55.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:54 smithi097 ceph-mon[24835]: from='client.24308 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi114=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:54:55.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:54 smithi097 ceph-mon[24835]: Saving service prometheus spec with placement smithi114=a;count:1 2024-04-12T21:54:55.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:54 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:55.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:54 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:54:55.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:54 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:54:55.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:54 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:54:55.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:54 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' 2024-04-12T21:54:55.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:54 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-12T21:54:55.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:54 smithi097 ceph-mon[24835]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-12T21:54:55.488 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:54:55 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: 2024-04-12T21:54:55.232+0000 7fd6ce6c4000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-12T21:54:55.488 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:54:55 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: 2024-04-12T21:54:55.328+0000 7fd6ce6c4000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-12T21:54:55.661 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:54:55 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:54:55.209+0000 7f2b13abc000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-12T21:54:55.661 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:54:55 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:54:55.297+0000 7f2b13abc000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-12T21:54:55.742 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:55 smithi114 ceph-mon[25759]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-12T21:54:55.742 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:54:55 smithi114 ceph-mon[25759]: mgrmap e16: y(active, since 4m), standbys: x 2024-04-12T21:54:55.968 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:55 smithi097 ceph-mon[24835]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-12T21:54:55.968 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:54:55 smithi097 ceph-mon[24835]: mgrmap e16: y(active, since 4m), standbys: x 2024-04-12T21:54:55.969 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:54:55 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:54:55.879+0000 7f2b13abc000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-12T21:54:55.969 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:55 smithi097 ceph-mon[29620]: from='mgr.14152 172.21.15.97:0/2938836947' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-12T21:54:55.969 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:54:55 smithi097 ceph-mon[29620]: mgrmap e16: y(active, since 4m), standbys: x 2024-04-12T21:54:55.994 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:54:55 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: 2024-04-12T21:54:55.946+0000 7fd6ce6c4000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-12T21:54:56.283 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:54:55 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:54:55.966+0000 7f2b13abc000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-12T21:54:56.283 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:54:56 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:54:56.053+0000 7f2b13abc000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-12T21:54:56.284 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:54:56 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:54:56.202+0000 7f2b13abc000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-12T21:54:56.284 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:54:56 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:54:56.281+0000 7f2b13abc000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-12T21:54:56.358 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:54:56 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: 2024-04-12T21:54:56.038+0000 7fd6ce6c4000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-12T21:54:56.358 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:54:56 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: 2024-04-12T21:54:56.124+0000 7fd6ce6c4000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-12T21:54:56.358 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:54:56 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: 2024-04-12T21:54:56.275+0000 7fd6ce6c4000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-12T21:54:56.358 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:54:56 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: 2024-04-12T21:54:56.355+0000 7fd6ce6c4000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-12T21:54:56.639 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:54:56 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:54:56.389+0000 7f2b13abc000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-12T21:54:56.639 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:54:56 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:54:56.472+0000 7f2b13abc000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-12T21:54:56.709 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:54:56 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: 2024-04-12T21:54:56.463+0000 7fd6ce6c4000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-12T21:54:56.710 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:54:56 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: 2024-04-12T21:54:56.545+0000 7fd6ce6c4000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-12T21:54:56.710 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:54:56 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: 2024-04-12T21:54:56.707+0000 7fd6ce6c4000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-12T21:54:56.911 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:54:56 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:54:56.641+0000 7f2b13abc000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-12T21:54:56.911 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:54:56 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:54:56.727+0000 7f2b13abc000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-12T21:54:56.994 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:54:56 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: 2024-04-12T21:54:56.791+0000 7fd6ce6c4000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-12T21:54:58.059 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:54:57 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:54:57.738+0000 7f2b13abc000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-12T21:54:58.059 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:54:57 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:54:57.867+0000 7f2b13abc000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-12T21:54:58.059 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:54:58 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:54:58.057+0000 7f2b13abc000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-12T21:54:58.116 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:54:57 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: 2024-04-12T21:54:57.801+0000 7fd6ce6c4000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-12T21:54:58.117 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:54:57 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: 2024-04-12T21:54:57.927+0000 7fd6ce6c4000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-12T21:54:58.411 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:54:58 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:54:58.144+0000 7f2b13abc000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-12T21:54:58.411 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:54:58 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:54:58.279+0000 7f2b13abc000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-12T21:54:58.494 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:54:58 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: 2024-04-12T21:54:58.114+0000 7fd6ce6c4000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-12T21:54:58.494 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:54:58 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: 2024-04-12T21:54:58.199+0000 7fd6ce6c4000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-12T21:54:58.494 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:54:58 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: 2024-04-12T21:54:58.331+0000 7fd6ce6c4000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-12T21:54:58.903 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:54:58 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:54:58.598+0000 7f2b13abc000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-12T21:54:58.903 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:54:58 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:54:58.818+0000 7f2b13abc000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-12T21:54:58.903 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:54:58 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:54:58.901+0000 7f2b13abc000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-12T21:54:58.946 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:54:58 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: 2024-04-12T21:54:58.646+0000 7fd6ce6c4000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-12T21:54:58.946 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:54:58 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: 2024-04-12T21:54:58.862+0000 7fd6ce6c4000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-12T21:54:59.161 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:54:59 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:54:59.071+0000 7f2b13abc000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-12T21:54:59.244 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:54:58 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: 2024-04-12T21:54:58.943+0000 7fd6ce6c4000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-12T21:54:59.244 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:54:59 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: 2024-04-12T21:54:59.110+0000 7fd6ce6c4000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-12T21:55:00.277 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:54:59 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:54:59.982+0000 7f2b13abc000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-12T21:55:00.278 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:55:00 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:55:00.276+0000 7f2b13abc000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-12T21:55:00.294 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:55:00 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: 2024-04-12T21:55:00.004+0000 7fd6ce6c4000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-12T21:55:00.584 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:55:00 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:55:00.372+0000 7f2b13abc000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-12T21:55:00.585 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:55:00 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:55:00.523+0000 7f2b13abc000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-12T21:55:00.585 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:55:00 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: 2024-04-12T21:55:00.292+0000 7fd6ce6c4000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-12T21:55:00.585 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:55:00 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: 2024-04-12T21:55:00.386+0000 7fd6ce6c4000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-12T21:55:00.585 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:55:00 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: 2024-04-12T21:55:00.533+0000 7fd6ce6c4000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-12T21:55:00.585 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:55:00 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: [12/Apr/2024:21:55:00] ENGINE Bus STARTING 2024-04-12T21:55:00.901 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:55:00 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: [12/Apr/2024:21:55:00] ENGINE Bus STARTING 2024-04-12T21:55:00.901 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:00 smithi097 ceph-mon[24835]: Active manager daemon y restarted 2024-04-12T21:55:00.901 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:00 smithi097 ceph-mon[24835]: Activating manager daemon y 2024-04-12T21:55:00.901 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:00 smithi097 ceph-mon[24835]: osdmap e59: 8 total, 8 up, 8 in 2024-04-12T21:55:00.901 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:00 smithi097 ceph-mon[24835]: Standby manager daemon x restarted 2024-04-12T21:55:00.902 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:00 smithi097 ceph-mon[24835]: Standby manager daemon x started 2024-04-12T21:55:00.902 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:00 smithi097 ceph-mon[24835]: from='mgr.? 172.21.15.114:0/1651996253' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-12T21:55:00.902 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:00 smithi097 ceph-mon[24835]: from='mgr.? 172.21.15.114:0/1651996253' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-12T21:55:00.902 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:00 smithi097 ceph-mon[24835]: from='mgr.? 172.21.15.114:0/1651996253' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-12T21:55:00.902 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:00 smithi097 ceph-mon[24835]: from='mgr.? 172.21.15.114:0/1651996253' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-12T21:55:00.902 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:00 smithi097 ceph-mon[29620]: Active manager daemon y restarted 2024-04-12T21:55:00.902 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:00 smithi097 ceph-mon[29620]: Activating manager daemon y 2024-04-12T21:55:00.902 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:00 smithi097 ceph-mon[29620]: osdmap e59: 8 total, 8 up, 8 in 2024-04-12T21:55:00.902 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:00 smithi097 ceph-mon[29620]: Standby manager daemon x restarted 2024-04-12T21:55:00.902 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:00 smithi097 ceph-mon[29620]: Standby manager daemon x started 2024-04-12T21:55:00.902 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:00 smithi097 ceph-mon[29620]: from='mgr.? 172.21.15.114:0/1651996253' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-12T21:55:00.903 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:00 smithi097 ceph-mon[29620]: from='mgr.? 172.21.15.114:0/1651996253' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-12T21:55:00.903 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:00 smithi097 ceph-mon[29620]: from='mgr.? 172.21.15.114:0/1651996253' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-12T21:55:00.903 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:00 smithi097 ceph-mon[29620]: from='mgr.? 172.21.15.114:0/1651996253' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-12T21:55:00.961 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:00 smithi114 ceph-mon[25759]: Active manager daemon y restarted 2024-04-12T21:55:00.961 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:00 smithi114 ceph-mon[25759]: Activating manager daemon y 2024-04-12T21:55:00.961 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:00 smithi114 ceph-mon[25759]: osdmap e59: 8 total, 8 up, 8 in 2024-04-12T21:55:00.961 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:00 smithi114 ceph-mon[25759]: Standby manager daemon x restarted 2024-04-12T21:55:00.961 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:00 smithi114 ceph-mon[25759]: Standby manager daemon x started 2024-04-12T21:55:00.961 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:00 smithi114 ceph-mon[25759]: from='mgr.? 172.21.15.114:0/1651996253' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-12T21:55:00.961 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:00 smithi114 ceph-mon[25759]: from='mgr.? 172.21.15.114:0/1651996253' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-12T21:55:00.961 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:00 smithi114 ceph-mon[25759]: from='mgr.? 172.21.15.114:0/1651996253' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-12T21:55:00.961 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:00 smithi114 ceph-mon[25759]: from='mgr.? 172.21.15.114:0/1651996253' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-12T21:55:00.962 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:55:00 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: CherryPy Checker: 2024-04-12T21:55:00.962 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:55:00 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: The Application mounted at '' has an empty config. 2024-04-12T21:55:00.962 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:55:00 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: 2024-04-12T21:55:00.962 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:55:00 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: [12/Apr/2024:21:55:00] ENGINE Serving on http://:::9283 2024-04-12T21:55:00.962 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:55:00 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: [12/Apr/2024:21:55:00] ENGINE Bus STARTED 2024-04-12T21:55:01.152 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:55:00 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: CherryPy Checker: 2024-04-12T21:55:01.153 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:55:00 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: The Application mounted at '' has an empty config. 2024-04-12T21:55:01.153 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:55:00 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T21:55:01.153 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:55:00 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: [12/Apr/2024:21:55:00] ENGINE Serving on http://:::9283 2024-04-12T21:55:01.153 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:55:00 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: [12/Apr/2024:21:55:00] ENGINE Bus STARTED 2024-04-12T21:55:01.153 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:55:01 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: [12/Apr/2024:21:55:01] ENGINE Bus STARTING 2024-04-12T21:55:01.411 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:55:01 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: [12/Apr/2024:21:55:01] ENGINE Serving on https://172.21.15.97:7150 2024-04-12T21:55:01.411 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:55:01 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: [12/Apr/2024:21:55:01] ENGINE Bus STARTED 2024-04-12T21:55:01.611 INFO:teuthology.orchestra.run.smithi114.stdout:Scheduled node-exporter update... 2024-04-12T21:55:01.859 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:01 smithi097 ceph-mon[24835]: mgrmap e17: y(active, starting, since 0.055879s), standbys: x 2024-04-12T21:55:01.859 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:01 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-12T21:55:01.859 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:01 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:55:01.859 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:01 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T21:55:01.859 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:01 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-12T21:55:01.859 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:01 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-12T21:55:01.859 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:01 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T21:55:01.859 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:01 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T21:55:01.859 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:01 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T21:55:01.859 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:01 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T21:55:01.859 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:01 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T21:55:01.859 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:01 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T21:55:01.859 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:01 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-12T21:55:01.860 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:01 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-12T21:55:01.860 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:01 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-12T21:55:01.860 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:01 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-12T21:55:01.860 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:01 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-12T21:55:01.860 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:01 smithi097 ceph-mon[24835]: Manager daemon y is now available 2024-04-12T21:55:01.860 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:01 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:55:01.860 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:01 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:55:01.860 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:01 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:55:01.860 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:01 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:55:01.860 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:01 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:55:01.860 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:01 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T21:55:01.860 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:01 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T21:55:01.860 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:01 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T21:55:01.861 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:01 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T21:55:01.861 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:01 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:55:01.861 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:01 smithi097 ceph-mon[29620]: mgrmap e17: y(active, starting, since 0.055879s), standbys: x 2024-04-12T21:55:01.861 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:01 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-12T21:55:01.861 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:01 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:55:01.861 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:01 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T21:55:01.861 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:01 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-12T21:55:01.861 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:01 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-12T21:55:01.861 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:01 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T21:55:01.861 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:01 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T21:55:01.861 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:01 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T21:55:01.861 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:01 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T21:55:01.862 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:01 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T21:55:01.862 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:01 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T21:55:01.862 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:01 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-12T21:55:01.862 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:01 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-12T21:55:01.862 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:01 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-12T21:55:01.862 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:01 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-12T21:55:01.862 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:01 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-12T21:55:01.862 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:01 smithi097 ceph-mon[29620]: Manager daemon y is now available 2024-04-12T21:55:01.862 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:01 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:55:01.862 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:01 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:55:01.862 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:01 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:55:01.862 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:01 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:55:01.862 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:01 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:55:01.862 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:01 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T21:55:01.863 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:01 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T21:55:01.863 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:01 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T21:55:01.863 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:01 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T21:55:01.863 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:01 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:55:01.894 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:01 smithi114 ceph-mon[25759]: mgrmap e17: y(active, starting, since 0.055879s), standbys: x 2024-04-12T21:55:01.894 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:01 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-12T21:55:01.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:01 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:55:01.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:01 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T21:55:01.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:01 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-12T21:55:01.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:01 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-12T21:55:01.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:01 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T21:55:01.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:01 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T21:55:01.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:01 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T21:55:01.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:01 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T21:55:01.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:01 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T21:55:01.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:01 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T21:55:01.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:01 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-12T21:55:01.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:01 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-12T21:55:01.895 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:01 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-12T21:55:01.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:01 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-12T21:55:01.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:01 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-12T21:55:01.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:01 smithi114 ceph-mon[25759]: Manager daemon y is now available 2024-04-12T21:55:01.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:01 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:55:01.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:01 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:55:01.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:01 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:55:01.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:01 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:55:01.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:01 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:55:01.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:01 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T21:55:01.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:01 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T21:55:01.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:01 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T21:55:01.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:01 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T21:55:01.896 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:01 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:55:02.236 DEBUG:teuthology.orchestra.run.smithi097:node-exporter.a> sudo journalctl -f -n 0 -u ceph-80bf7680-f916-11ee-bc8d-c7b262605968@node-exporter.a.service 2024-04-12T21:55:02.238 DEBUG:teuthology.orchestra.run.smithi114:node-exporter.b> sudo journalctl -f -n 0 -u ceph-80bf7680-f916-11ee-bc8d-c7b262605968@node-exporter.b.service 2024-04-12T21:55:02.241 INFO:tasks.cephadm:Adding alertmanager.a on smithi097 2024-04-12T21:55:02.241 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- ceph orch apply alertmanager '1;smithi097=a' 2024-04-12T21:55:02.842 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:02 smithi114 ceph-mon[25759]: [12/Apr/2024:21:55:01] ENGINE Bus STARTING 2024-04-12T21:55:02.843 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:02 smithi114 ceph-mon[25759]: [12/Apr/2024:21:55:01] ENGINE Serving on https://172.21.15.97:7150 2024-04-12T21:55:02.843 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:02 smithi114 ceph-mon[25759]: [12/Apr/2024:21:55:01] ENGINE Bus STARTED 2024-04-12T21:55:02.843 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:02 smithi114 ceph-mon[25759]: mgrmap e18: y(active, since 1.07341s), standbys: x 2024-04-12T21:55:02.843 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:02 smithi114 ceph-mon[25759]: from='client.24326 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi097=a;smithi114=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:55:02.843 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:02 smithi114 ceph-mon[25759]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:02.843 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:02 smithi114 ceph-mon[25759]: Saving service node-exporter spec with placement smithi097=a;smithi114=b;count:2 2024-04-12T21:55:02.843 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:02 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:55:02.843 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:02 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:55:02.905 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:02 smithi097 ceph-mon[24835]: [12/Apr/2024:21:55:01] ENGINE Bus STARTING 2024-04-12T21:55:02.905 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:02 smithi097 ceph-mon[24835]: [12/Apr/2024:21:55:01] ENGINE Serving on https://172.21.15.97:7150 2024-04-12T21:55:02.905 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:02 smithi097 ceph-mon[24835]: [12/Apr/2024:21:55:01] ENGINE Bus STARTED 2024-04-12T21:55:02.905 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:02 smithi097 ceph-mon[24835]: mgrmap e18: y(active, since 1.07341s), standbys: x 2024-04-12T21:55:02.905 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:02 smithi097 ceph-mon[24835]: from='client.24326 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi097=a;smithi114=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:55:02.905 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:02 smithi097 ceph-mon[24835]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:02.905 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:02 smithi097 ceph-mon[24835]: Saving service node-exporter spec with placement smithi097=a;smithi114=b;count:2 2024-04-12T21:55:02.905 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:02 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:55:02.905 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:02 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:55:02.906 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:02 smithi097 ceph-mon[29620]: [12/Apr/2024:21:55:01] ENGINE Bus STARTING 2024-04-12T21:55:02.906 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:02 smithi097 ceph-mon[29620]: [12/Apr/2024:21:55:01] ENGINE Serving on https://172.21.15.97:7150 2024-04-12T21:55:02.906 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:02 smithi097 ceph-mon[29620]: [12/Apr/2024:21:55:01] ENGINE Bus STARTED 2024-04-12T21:55:02.906 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:02 smithi097 ceph-mon[29620]: mgrmap e18: y(active, since 1.07341s), standbys: x 2024-04-12T21:55:02.906 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:02 smithi097 ceph-mon[29620]: from='client.24326 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi097=a;smithi114=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:55:02.906 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:02 smithi097 ceph-mon[29620]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:02.906 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:02 smithi097 ceph-mon[29620]: Saving service node-exporter spec with placement smithi097=a;smithi114=b;count:2 2024-04-12T21:55:02.906 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:02 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:55:02.906 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:02 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:55:03.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:03 smithi097 ceph-mon[24835]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:03.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:03 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:55:03.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:03 smithi097 ceph-mon[24835]: mgrmap e19: y(active, since 2s), standbys: x 2024-04-12T21:55:03.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:03 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:55:03.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:03 smithi097 ceph-mon[29620]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:03.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:03 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:55:03.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:03 smithi097 ceph-mon[29620]: mgrmap e19: y(active, since 2s), standbys: x 2024-04-12T21:55:03.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:03 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:55:03.870 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:03 smithi114 ceph-mon[25759]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:03.870 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:03 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:55:03.870 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:03 smithi114 ceph-mon[25759]: mgrmap e19: y(active, since 2s), standbys: x 2024-04-12T21:55:03.870 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:03 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:55:04.403 INFO:teuthology.orchestra.run.smithi114.stdout:Scheduled alertmanager update... 2024-04-12T21:55:04.870 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:04 smithi114 ceph-mon[25759]: Updating smithi097:/etc/ceph/ceph.conf 2024-04-12T21:55:04.870 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:04 smithi114 ceph-mon[25759]: Updating smithi097:/etc/ceph/ceph.client.admin.keyring 2024-04-12T21:55:04.870 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:04 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:55:04.870 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:04 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:55:04.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:04 smithi097 ceph-mon[24835]: Updating smithi097:/etc/ceph/ceph.conf 2024-04-12T21:55:04.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:04 smithi097 ceph-mon[24835]: Updating smithi097:/etc/ceph/ceph.client.admin.keyring 2024-04-12T21:55:04.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:04 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:55:04.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:04 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:55:04.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:04 smithi097 ceph-mon[29620]: Updating smithi097:/etc/ceph/ceph.conf 2024-04-12T21:55:04.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:04 smithi097 ceph-mon[29620]: Updating smithi097:/etc/ceph/ceph.client.admin.keyring 2024-04-12T21:55:04.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:04 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:55:04.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:04 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:55:05.059 DEBUG:teuthology.orchestra.run.smithi097:alertmanager.a> sudo journalctl -f -n 0 -u ceph-80bf7680-f916-11ee-bc8d-c7b262605968@alertmanager.a.service 2024-04-12T21:55:05.061 INFO:tasks.cephadm:Adding grafana.a on smithi114 2024-04-12T21:55:05.062 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- ceph orch apply grafana '1;smithi114=a' 2024-04-12T21:55:05.658 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:05 smithi114 ceph-mon[25759]: from='client.24347 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi097=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:55:05.659 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:05 smithi114 ceph-mon[25759]: Saving service alertmanager spec with placement smithi097=a;count:1 2024-04-12T21:55:05.659 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:05 smithi114 ceph-mon[25759]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:05.659 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:05 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:55:05.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:05 smithi097 ceph-mon[24835]: from='client.24347 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi097=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:55:05.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:05 smithi097 ceph-mon[24835]: Saving service alertmanager spec with placement smithi097=a;count:1 2024-04-12T21:55:05.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:05 smithi097 ceph-mon[24835]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:05.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:05 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:55:05.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:05 smithi097 ceph-mon[29620]: from='client.24347 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi097=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:55:05.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:05 smithi097 ceph-mon[29620]: Saving service alertmanager spec with placement smithi097=a;count:1 2024-04-12T21:55:05.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:05 smithi097 ceph-mon[29620]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:05.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:05 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:55:06.903 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:06 smithi114 ceph-mon[25759]: Updating smithi114:/etc/ceph/ceph.conf 2024-04-12T21:55:06.903 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:06 smithi114 ceph-mon[25759]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-04-12T21:55:06.903 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:06 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:55:06.903 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:06 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:55:06.903 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:06 smithi114 ceph-mon[25759]: Deploying daemon prometheus.a on smithi114 2024-04-12T21:55:06.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:06 smithi097 ceph-mon[24835]: Updating smithi114:/etc/ceph/ceph.conf 2024-04-12T21:55:06.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:06 smithi097 ceph-mon[24835]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-04-12T21:55:06.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:06 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:55:06.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:06 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:55:06.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:06 smithi097 ceph-mon[24835]: Deploying daemon prometheus.a on smithi114 2024-04-12T21:55:06.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:06 smithi097 ceph-mon[29620]: Updating smithi114:/etc/ceph/ceph.conf 2024-04-12T21:55:06.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:06 smithi097 ceph-mon[29620]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-04-12T21:55:06.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:06 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:55:06.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:06 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:55:06.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:06 smithi097 ceph-mon[29620]: Deploying daemon prometheus.a on smithi114 2024-04-12T21:55:07.066 INFO:teuthology.orchestra.run.smithi114.stdout:Scheduled grafana update... 2024-04-12T21:55:07.675 DEBUG:teuthology.orchestra.run.smithi114:grafana.a> sudo journalctl -f -n 0 -u ceph-80bf7680-f916-11ee-bc8d-c7b262605968@grafana.a.service 2024-04-12T21:55:07.678 INFO:tasks.cephadm:Setting up client nodes... 2024-04-12T21:55:07.681 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-12T21:55:07.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:07 smithi097 ceph-mon[24835]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:07.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:07 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:55:07.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:07 smithi097 ceph-mon[29620]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:07.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:07 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:55:07.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:07 smithi114 ceph-mon[25759]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:07.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:07 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:55:08.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:08 smithi097 ceph-mon[24835]: from='client.24353 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi114=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:55:08.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:08 smithi097 ceph-mon[24835]: Saving service grafana spec with placement smithi114=a;count:1 2024-04-12T21:55:08.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:08 smithi097 ceph-mon[29620]: from='client.24353 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi114=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:55:08.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:08 smithi097 ceph-mon[29620]: Saving service grafana spec with placement smithi114=a;count:1 2024-04-12T21:55:08.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:08 smithi114 ceph-mon[25759]: from='client.24353 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi114=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:55:08.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:08 smithi114 ceph-mon[25759]: Saving service grafana spec with placement smithi114=a;count:1 2024-04-12T21:55:09.843 INFO:teuthology.orchestra.run.smithi097.stdout:[client.0] 2024-04-12T21:55:09.843 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQC9rRlms9L8MRAAZkzuXBUROsvZPuYBMnybmg== 2024-04-12T21:55:09.844 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:09 smithi097 ceph-mon[24835]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:09.844 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:09 smithi097 ceph-mon[29620]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:09.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:09 smithi114 ceph-mon[25759]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:10.434 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-04-12T21:55:10.435 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-12T21:55:10.435 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-12T21:55:10.480 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-12T21:55:10.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:10 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/247226329' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-12T21:55:10.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:10 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/247226329' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-12T21:55:10.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:10 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/247226329' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-12T21:55:10.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:10 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/247226329' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-12T21:55:10.784 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:10 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/247226329' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-12T21:55:10.784 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:10 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/247226329' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-12T21:55:11.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:11 smithi097 ceph-mon[24835]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:11.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:11 smithi097 ceph-mon[29620]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:11.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:11 smithi114 ceph-mon[25759]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:13.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:13 smithi097 ceph-mon[24835]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:13.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:13 smithi097 ceph-mon[29620]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:13.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:13 smithi114 ceph-mon[25759]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:14.665 INFO:teuthology.orchestra.run.smithi114.stdout:[client.1] 2024-04-12T21:55:14.665 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQDCrRlmXBtiJxAA5VbgKSWUOwMAeUdEw6PnRw== 2024-04-12T21:55:15.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:15 smithi097 ceph-mon[24835]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:15.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:15 smithi097 ceph-mon[24835]: from='client.? 172.21.15.114:0/2021207165' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-12T21:55:15.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:15 smithi097 ceph-mon[24835]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-12T21:55:15.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:15 smithi097 ceph-mon[24835]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-12T21:55:15.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:15 smithi097 ceph-mon[29620]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:15.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:15 smithi097 ceph-mon[29620]: from='client.? 172.21.15.114:0/2021207165' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-12T21:55:15.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:15 smithi097 ceph-mon[29620]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-12T21:55:15.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:15 smithi097 ceph-mon[29620]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-12T21:55:15.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:15 smithi114 ceph-mon[25759]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:15.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:15 smithi114 ceph-mon[25759]: from='client.? 172.21.15.114:0/2021207165' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-12T21:55:15.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:15 smithi114 ceph-mon[25759]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-12T21:55:15.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:15 smithi114 ceph-mon[25759]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-12T21:55:16.495 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-04-12T21:55:16.495 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-04-12T21:55:16.495 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-12T21:55:16.542 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-12T21:55:16.542 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-12T21:55:16.542 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- ceph mgr dump --format=json 2024-04-12T21:55:16.765 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:55:17.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:17 smithi097 ceph-mon[29620]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:17.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:17 smithi097 ceph-mon[24835]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:17.952 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:17 smithi114 ceph-mon[25759]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:18.629 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:55:18 smithi114 systemd[1]: Starting Ceph prometheus.a for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T21:55:18.695 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-12T21:55:18.898 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:55:18 smithi114 podman[40845]: 2024-04-12 21:55:18.531490826 +0000 UTC m=+0.018633613 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-04-12T21:55:18.898 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:55:18 smithi114 podman[40845]: 2024-04-12 21:55:18.653410226 +0000 UTC m=+0.140553055 container create df709a0468dddca7d46706b76c467f1ca18faeb24cd18f6b67f70edd1ca7fd35 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-12T21:55:18.898 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:55:18 smithi114 podman[40845]: 2024-04-12 21:55:18.897181872 +0000 UTC m=+0.384324659 container init df709a0468dddca7d46706b76c467f1ca18faeb24cd18f6b67f70edd1ca7fd35 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-12T21:55:19.244 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:55:18 smithi114 podman[40845]: 2024-04-12 21:55:18.899785267 +0000 UTC m=+0.386928064 container start df709a0468dddca7d46706b76c467f1ca18faeb24cd18f6b67f70edd1ca7fd35 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-12T21:55:19.245 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:55:18 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[40860]: ts=2024-04-12T21:55:18.923Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-04-12T21:55:19.245 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:55:18 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[40860]: ts=2024-04-12T21:55:18.923Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-04-12T21:55:19.245 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:55:18 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[40860]: ts=2024-04-12T21:55:18.923Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-04-12T21:55:19.245 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:55:18 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[40860]: ts=2024-04-12T21:55:18.923Z caller=main.go:518 level=info host_details="(Linux 5.14.0-435.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024 x86_64 smithi114 (none))" 2024-04-12T21:55:19.245 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:55:18 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[40860]: ts=2024-04-12T21:55:18.923Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-04-12T21:55:19.245 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:55:18 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[40860]: ts=2024-04-12T21:55:18.923Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-04-12T21:55:19.245 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:55:18 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[40860]: ts=2024-04-12T21:55:18.925Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-04-12T21:55:19.245 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:55:18 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[40860]: ts=2024-04-12T21:55:18.925Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-04-12T21:55:19.245 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:55:18 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[40860]: ts=2024-04-12T21:55:18.926Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-04-12T21:55:19.245 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:55:18 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[40860]: ts=2024-04-12T21:55:18.927Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-04-12T21:55:19.245 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:55:18 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[40860]: ts=2024-04-12T21:55:18.927Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.681µs 2024-04-12T21:55:19.245 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:55:18 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[40860]: ts=2024-04-12T21:55:18.927Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-04-12T21:55:19.246 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:55:18 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[40860]: ts=2024-04-12T21:55:18.927Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-04-12T21:55:19.246 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:55:18 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[40860]: ts=2024-04-12T21:55:18.927Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=32.502µs wal_replay_duration=182.664µs total_replay_duration=228.237µs 2024-04-12T21:55:19.246 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:55:18 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[40860]: ts=2024-04-12T21:55:18.928Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-04-12T21:55:19.246 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:55:18 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[40860]: ts=2024-04-12T21:55:18.928Z caller=main.go:947 level=info msg="TSDB started" 2024-04-12T21:55:19.246 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:55:18 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[40860]: ts=2024-04-12T21:55:18.928Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-04-12T21:55:19.246 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:55:18 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[40860]: ts=2024-04-12T21:55:18.944Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=15.825939ms db_storage=836ns remote_storage=1.782µs web_handler=161ns query_engine=582ns scrape=827.822µs scrape_sd=15.05µs notify=757ns notify_sd=1.229µs rules=14.781648ms 2024-04-12T21:55:19.246 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:55:18 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[40860]: ts=2024-04-12T21:55:18.944Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-04-12T21:55:19.246 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:55:18 smithi114 bash[40845]: df709a0468dddca7d46706b76c467f1ca18faeb24cd18f6b67f70edd1ca7fd35 2024-04-12T21:55:19.246 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:55:18 smithi114 systemd[1]: Started Ceph prometheus.a for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T21:55:19.388 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":19,"active_gid":24352,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":560924382},{"type":"v1","addr":"172.21.15.97:6801","nonce":560924382}]},"active_addr":"172.21.15.97:6801/560924382","active_change":"2024-04-12T21:55:00.526329+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[{"gid":24314,"name":"x","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.97:8443/","prometheus":"http://172.21.15.97:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":59,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.97:0","nonce":1069516662}]},{"addrvec":[{"type":"v2","addr":"172.21.15.97:0","nonce":652135493}]},{"addrvec":[{"type":"v2","addr":"172.21.15.97:0","nonce":903882336}]},{"addrvec":[{"type":"v2","addr":"172.21.15.97:0","nonce":2307085899}]}]}} 2024-04-12T21:55:19.391 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-12T21:55:19.391 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-12T21:55:19.391 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- ceph osd dump --format=json 2024-04-12T21:55:19.630 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:55:19.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:19 smithi097 ceph-mon[24835]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:19.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:19 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/2569001724' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-12T21:55:19.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:19 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:55:19.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:19 smithi097 ceph-mon[29620]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:19.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:19 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/2569001724' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-12T21:55:19.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:19 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:55:19.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:19 smithi114 ceph-mon[25759]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:19.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:19 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/2569001724' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-12T21:55:19.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:19 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:55:20.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:20 smithi097 ceph-mon[24835]: Deploying daemon node-exporter.a on smithi097 2024-04-12T21:55:20.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:20 smithi097 ceph-mon[29620]: Deploying daemon node-exporter.a on smithi097 2024-04-12T21:55:20.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:20 smithi114 ceph-mon[25759]: Deploying daemon node-exporter.a on smithi097 2024-04-12T21:55:21.548 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-12T21:55:21.548 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":59,"fsid":"80bf7680-f916-11ee-bc8d-c7b262605968","created":"2024-04-12T21:49:48.426985+0000","modified":"2024-04-12T21:55:00.526020+0000","last_up_change":"2024-04-12T21:54:44.911811+0000","last_in_change":"2024-04-12T21:54:28.414803+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-12T21:52:54.377978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"315f320f-95fd-4b18-ab6a-417ba7651667","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":2413859098},{"type":"v1","addr":"172.21.15.97:6803","nonce":2413859098}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":2413859098},{"type":"v1","addr":"172.21.15.97:6805","nonce":2413859098}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2413859098},{"type":"v1","addr":"172.21.15.97:6809","nonce":2413859098}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":2413859098},{"type":"v1","addr":"172.21.15.97:6807","nonce":2413859098}]},"public_addr":"172.21.15.97:6803/2413859098","cluster_addr":"172.21.15.97:6805/2413859098","heartbeat_back_addr":"172.21.15.97:6809/2413859098","heartbeat_front_addr":"172.21.15.97:6807/2413859098","state":["exists","up"]},{"osd":1,"uuid":"c58dc94b-86d4-4b83-95ad-d3779a5aa46e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":1786985012},{"type":"v1","addr":"172.21.15.97:6811","nonce":1786985012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":1786985012},{"type":"v1","addr":"172.21.15.97:6813","nonce":1786985012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":1786985012},{"type":"v1","addr":"172.21.15.97:6817","nonce":1786985012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":1786985012},{"type":"v1","addr":"172.21.15.97:6815","nonce":1786985012}]},"public_addr":"172.21.15.97:6811/1786985012","cluster_addr":"172.21.15.97:6813/1786985012","heartbeat_back_addr":"172.21.15.97:6817/1786985012","heartbeat_front_addr":"172.21.15.97:6815/1786985012","state":["exists","up"]},{"osd":2,"uuid":"4920ee2a-de4d-4072-92bb-dae616a76152","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":1944472297},{"type":"v1","addr":"172.21.15.97:6819","nonce":1944472297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":1944472297},{"type":"v1","addr":"172.21.15.97:6821","nonce":1944472297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1944472297},{"type":"v1","addr":"172.21.15.97:6825","nonce":1944472297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":1944472297},{"type":"v1","addr":"172.21.15.97:6823","nonce":1944472297}]},"public_addr":"172.21.15.97:6819/1944472297","cluster_addr":"172.21.15.97:6821/1944472297","heartbeat_back_addr":"172.21.15.97:6825/1944472297","heartbeat_front_addr":"172.21.15.97:6823/1944472297","state":["exists","up"]},{"osd":3,"uuid":"928b3bd7-c0df-443c-9a28-5ceb62ae3160","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":75554646},{"type":"v1","addr":"172.21.15.97:6827","nonce":75554646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":75554646},{"type":"v1","addr":"172.21.15.97:6829","nonce":75554646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":75554646},{"type":"v1","addr":"172.21.15.97:6833","nonce":75554646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":75554646},{"type":"v1","addr":"172.21.15.97:6831","nonce":75554646}]},"public_addr":"172.21.15.97:6827/75554646","cluster_addr":"172.21.15.97:6829/75554646","heartbeat_back_addr":"172.21.15.97:6833/75554646","heartbeat_front_addr":"172.21.15.97:6831/75554646","state":["exists","up"]},{"osd":4,"uuid":"02f49773-bcfc-499c-9174-d9cb5991c789","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":2190104221},{"type":"v1","addr":"172.21.15.114:6801","nonce":2190104221}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":2190104221},{"type":"v1","addr":"172.21.15.114:6803","nonce":2190104221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":2190104221},{"type":"v1","addr":"172.21.15.114:6807","nonce":2190104221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":2190104221},{"type":"v1","addr":"172.21.15.114:6805","nonce":2190104221}]},"public_addr":"172.21.15.114:6801/2190104221","cluster_addr":"172.21.15.114:6803/2190104221","heartbeat_back_addr":"172.21.15.114:6807/2190104221","heartbeat_front_addr":"172.21.15.114:6805/2190104221","state":["exists","up"]},{"osd":5,"uuid":"997405e6-e6f2-4c40-b81c-7848c56c062c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":515183706},{"type":"v1","addr":"172.21.15.114:6809","nonce":515183706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":515183706},{"type":"v1","addr":"172.21.15.114:6811","nonce":515183706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":515183706},{"type":"v1","addr":"172.21.15.114:6815","nonce":515183706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":515183706},{"type":"v1","addr":"172.21.15.114:6813","nonce":515183706}]},"public_addr":"172.21.15.114:6809/515183706","cluster_addr":"172.21.15.114:6811/515183706","heartbeat_back_addr":"172.21.15.114:6815/515183706","heartbeat_front_addr":"172.21.15.114:6813/515183706","state":["exists","up"]},{"osd":6,"uuid":"68b36dad-0aff-40a4-b8f7-8c1dcf42a5c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":2512498411},{"type":"v1","addr":"172.21.15.114:6817","nonce":2512498411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":2512498411},{"type":"v1","addr":"172.21.15.114:6819","nonce":2512498411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":2512498411},{"type":"v1","addr":"172.21.15.114:6823","nonce":2512498411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":2512498411},{"type":"v1","addr":"172.21.15.114:6821","nonce":2512498411}]},"public_addr":"172.21.15.114:6817/2512498411","cluster_addr":"172.21.15.114:6819/2512498411","heartbeat_back_addr":"172.21.15.114:6823/2512498411","heartbeat_front_addr":"172.21.15.114:6821/2512498411","state":["exists","up"]},{"osd":7,"uuid":"719dfe0e-c27b-4ada-a3e0-e381d70eb53b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":1147837664},{"type":"v1","addr":"172.21.15.114:6825","nonce":1147837664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":1147837664},{"type":"v1","addr":"172.21.15.114:6827","nonce":1147837664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":1147837664},{"type":"v1","addr":"172.21.15.114:6831","nonce":1147837664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":1147837664},{"type":"v1","addr":"172.21.15.114:6829","nonce":1147837664}]},"public_addr":"172.21.15.114:6825/1147837664","cluster_addr":"172.21.15.114:6827/1147837664","heartbeat_back_addr":"172.21.15.114:6831/1147837664","heartbeat_front_addr":"172.21.15.114:6829/1147837664","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:52:02.934510+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:52:26.942832+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:52:51.162399+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:53:15.754757+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:53:36.448575+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:53:58.560201+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:54:20.557106+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:54:43.500361+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.97:0/2357886921":"2024-04-13T21:55:00.525973+0000","172.21.15.97:6801/2093806388":"2024-04-13T21:55:00.525973+0000","172.21.15.97:6800/2093806388":"2024-04-13T21:55:00.525973+0000","172.21.15.97:0/2789980025":"2024-04-13T21:55:00.525973+0000","172.21.15.97:0/1481749366":"2024-04-13T21:55:00.525973+0000","172.21.15.97:0/831596468":"2024-04-13T21:50:43.236028+0000","172.21.15.97:0/1510317264":"2024-04-13T21:55:00.525973+0000","172.21.15.97:0/1562080679":"2024-04-13T21:50:43.236028+0000","172.21.15.97:6801/258588252":"2024-04-13T21:50:43.236028+0000","172.21.15.97:0/2696421935":"2024-04-13T21:50:43.236028+0000","172.21.15.97:6800/258588252":"2024-04-13T21:50:43.236028+0000","172.21.15.97:0/2460245404":"2024-04-13T21:50:20.870746+0000","172.21.15.97:0/1828347223":"2024-04-13T21:50:20.870746+0000","172.21.15.97:6800/1796168157":"2024-04-13T21:50:20.870746+0000","172.21.15.97:6801/1796168157":"2024-04-13T21:50:20.870746+0000","172.21.15.97:0/1917553394":"2024-04-13T21:50:20.870746+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-12T21:55:21.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:21 smithi097 ceph-mon[24835]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:21.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:21 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:55:21.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:21 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/2261385718' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:55:21.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:21 smithi097 ceph-mon[29620]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:21.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:21 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:55:21.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:21 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/2261385718' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:55:21.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:21 smithi114 ceph-mon[25759]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:21.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:21 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:55:21.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:21 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/2261385718' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:55:23.083 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-12T21:55:23.084 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- ceph osd dump --format=json 2024-04-12T21:55:23.411 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 21:55:23 smithi097 bash[44462]: Writing manifest to image destination 2024-04-12T21:55:23.525 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:55:23.825 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:23 smithi097 ceph-mon[24835]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:23.825 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:23 smithi097 ceph-mon[29620]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:23.825 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 21:55:23 smithi097 podman[44462]: 2024-04-12 21:55:23.515869161 +0000 UTC m=+3.137531159 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-04-12T21:55:23.825 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 21:55:23 smithi097 podman[44462]: 2024-04-12 21:55:23.605392027 +0000 UTC m=+3.227054006 container create 265c1a99d585808ff27e1ab28c03f0ddb12ac6d385ab1e487c9dcc1d9101c880 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-12T21:55:23.825 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 21:55:23 smithi097 podman[44462]: 2024-04-12 21:55:23.824164166 +0000 UTC m=+3.445826142 container init 265c1a99d585808ff27e1ab28c03f0ddb12ac6d385ab1e487c9dcc1d9101c880 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-12T21:55:23.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:23 smithi114 ceph-mon[25759]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:24.127 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 21:55:23 smithi097 podman[44462]: 2024-04-12 21:55:23.826910933 +0000 UTC m=+3.448572914 container start 265c1a99d585808ff27e1ab28c03f0ddb12ac6d385ab1e487c9dcc1d9101c880 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-12T21:55:24.261 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 21:55:23 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[44638]: ts=2024-04-12T21:55:23.830Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-04-12T21:55:24.261 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 21:55:23 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[44638]: ts=2024-04-12T21:55:23.830Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-04-12T21:55:24.261 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 21:55:23 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[44638]: ts=2024-04-12T21:55:23.831Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-04-12T21:55:24.261 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 21:55:23 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[44638]: ts=2024-04-12T21:55:23.831Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-04-12T21:55:24.261 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 21:55:23 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[44638]: ts=2024-04-12T21:55:23.831Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-04-12T21:55:24.261 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 21:55:23 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[44638]: ts=2024-04-12T21:55:23.831Z caller=node_exporter.go:115 level=info collector=arp 2024-04-12T21:55:24.261 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 21:55:23 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[44638]: ts=2024-04-12T21:55:23.831Z caller=node_exporter.go:115 level=info collector=bcache 2024-04-12T21:55:24.262 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 21:55:23 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[44638]: ts=2024-04-12T21:55:23.831Z caller=node_exporter.go:115 level=info collector=bonding 2024-04-12T21:55:24.262 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 21:55:23 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[44638]: ts=2024-04-12T21:55:23.831Z caller=node_exporter.go:115 level=info collector=btrfs 2024-04-12T21:55:24.262 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 21:55:23 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[44638]: ts=2024-04-12T21:55:23.831Z caller=node_exporter.go:115 level=info collector=conntrack 2024-04-12T21:55:24.262 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 21:55:23 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[44638]: ts=2024-04-12T21:55:23.831Z caller=node_exporter.go:115 level=info collector=cpu 2024-04-12T21:55:24.262 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 21:55:23 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[44638]: ts=2024-04-12T21:55:23.831Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-04-12T21:55:24.262 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 21:55:23 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[44638]: ts=2024-04-12T21:55:23.831Z caller=node_exporter.go:115 level=info collector=diskstats 2024-04-12T21:55:24.262 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 21:55:23 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[44638]: ts=2024-04-12T21:55:23.831Z caller=node_exporter.go:115 level=info collector=dmi 2024-04-12T21:55:24.262 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 21:55:23 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[44638]: ts=2024-04-12T21:55:23.831Z caller=node_exporter.go:115 level=info collector=edac 2024-04-12T21:55:24.262 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 21:55:23 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[44638]: ts=2024-04-12T21:55:23.831Z caller=node_exporter.go:115 level=info collector=entropy 2024-04-12T21:55:24.262 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 21:55:23 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[44638]: ts=2024-04-12T21:55:23.831Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-04-12T21:55:24.262 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 21:55:23 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[44638]: ts=2024-04-12T21:55:23.831Z caller=node_exporter.go:115 level=info collector=filefd 2024-04-12T21:55:24.262 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 21:55:23 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[44638]: ts=2024-04-12T21:55:23.831Z caller=node_exporter.go:115 level=info collector=filesystem 2024-04-12T21:55:24.262 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 21:55:23 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[44638]: ts=2024-04-12T21:55:23.831Z caller=node_exporter.go:115 level=info collector=hwmon 2024-04-12T21:55:24.263 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 21:55:23 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[44638]: ts=2024-04-12T21:55:23.831Z caller=node_exporter.go:115 level=info collector=infiniband 2024-04-12T21:55:24.263 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 21:55:23 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[44638]: ts=2024-04-12T21:55:23.831Z caller=node_exporter.go:115 level=info collector=ipvs 2024-04-12T21:55:24.263 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 21:55:23 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[44638]: ts=2024-04-12T21:55:23.831Z caller=node_exporter.go:115 level=info collector=loadavg 2024-04-12T21:55:24.263 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 21:55:23 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[44638]: ts=2024-04-12T21:55:23.831Z caller=node_exporter.go:115 level=info collector=mdadm 2024-04-12T21:55:24.263 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 21:55:23 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[44638]: ts=2024-04-12T21:55:23.831Z caller=node_exporter.go:115 level=info collector=meminfo 2024-04-12T21:55:24.263 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 21:55:23 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[44638]: ts=2024-04-12T21:55:23.831Z caller=node_exporter.go:115 level=info collector=netclass 2024-04-12T21:55:24.263 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 21:55:23 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[44638]: ts=2024-04-12T21:55:23.831Z caller=node_exporter.go:115 level=info collector=netdev 2024-04-12T21:55:24.263 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 21:55:23 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[44638]: ts=2024-04-12T21:55:23.831Z caller=node_exporter.go:115 level=info collector=netstat 2024-04-12T21:55:24.263 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 21:55:23 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[44638]: ts=2024-04-12T21:55:23.831Z caller=node_exporter.go:115 level=info collector=nfs 2024-04-12T21:55:24.263 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 21:55:23 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[44638]: ts=2024-04-12T21:55:23.831Z caller=node_exporter.go:115 level=info collector=nfsd 2024-04-12T21:55:24.263 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 21:55:23 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[44638]: ts=2024-04-12T21:55:23.831Z caller=node_exporter.go:115 level=info collector=nvme 2024-04-12T21:55:24.263 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 21:55:23 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[44638]: ts=2024-04-12T21:55:23.831Z caller=node_exporter.go:115 level=info collector=os 2024-04-12T21:55:24.263 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 21:55:23 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[44638]: ts=2024-04-12T21:55:23.831Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-04-12T21:55:24.263 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 21:55:23 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[44638]: ts=2024-04-12T21:55:23.831Z caller=node_exporter.go:115 level=info collector=pressure 2024-04-12T21:55:24.264 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 21:55:23 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[44638]: ts=2024-04-12T21:55:23.831Z caller=node_exporter.go:115 level=info collector=rapl 2024-04-12T21:55:24.264 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 21:55:23 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[44638]: ts=2024-04-12T21:55:23.831Z caller=node_exporter.go:115 level=info collector=schedstat 2024-04-12T21:55:24.264 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 21:55:23 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[44638]: ts=2024-04-12T21:55:23.831Z caller=node_exporter.go:115 level=info collector=sockstat 2024-04-12T21:55:24.264 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 21:55:23 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[44638]: ts=2024-04-12T21:55:23.831Z caller=node_exporter.go:115 level=info collector=softnet 2024-04-12T21:55:24.264 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 21:55:23 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[44638]: ts=2024-04-12T21:55:23.831Z caller=node_exporter.go:115 level=info collector=stat 2024-04-12T21:55:24.264 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 21:55:23 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[44638]: ts=2024-04-12T21:55:23.831Z caller=node_exporter.go:115 level=info collector=tapestats 2024-04-12T21:55:24.264 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 21:55:23 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[44638]: ts=2024-04-12T21:55:23.831Z caller=node_exporter.go:115 level=info collector=textfile 2024-04-12T21:55:24.265 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 21:55:23 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[44638]: ts=2024-04-12T21:55:23.831Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-04-12T21:55:24.265 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 21:55:23 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[44638]: ts=2024-04-12T21:55:23.832Z caller=node_exporter.go:115 level=info collector=time 2024-04-12T21:55:24.265 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 21:55:23 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[44638]: ts=2024-04-12T21:55:23.832Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-04-12T21:55:24.265 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 21:55:23 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[44638]: ts=2024-04-12T21:55:23.832Z caller=node_exporter.go:115 level=info collector=uname 2024-04-12T21:55:24.265 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 21:55:23 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[44638]: ts=2024-04-12T21:55:23.832Z caller=node_exporter.go:115 level=info collector=vmstat 2024-04-12T21:55:24.265 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 21:55:23 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[44638]: ts=2024-04-12T21:55:23.832Z caller=node_exporter.go:115 level=info collector=xfs 2024-04-12T21:55:24.265 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 21:55:23 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[44638]: ts=2024-04-12T21:55:23.832Z caller=node_exporter.go:115 level=info collector=zfs 2024-04-12T21:55:24.265 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 21:55:23 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[44638]: ts=2024-04-12T21:55:23.832Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-04-12T21:55:24.265 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 21:55:23 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[44638]: ts=2024-04-12T21:55:23.832Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-04-12T21:55:24.265 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 21:55:23 smithi097 bash[44462]: 265c1a99d585808ff27e1ab28c03f0ddb12ac6d385ab1e487c9dcc1d9101c880 2024-04-12T21:55:24.265 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 21:55:23 smithi097 systemd[1]: Started Ceph node-exporter.a for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T21:55:24.643 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:55:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: ::ffff:172.21.15.114 - - [12/Apr/2024:21:55:24] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-12T21:55:24.643 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:24 smithi114 systemd[1]: Starting Ceph node-exporter.b for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T21:55:24.977 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:24 smithi114 bash[41087]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-04-12T21:55:25.234 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:24 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:55:25.235 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:24 smithi097 ceph-mon[24835]: Deploying daemon node-exporter.b on smithi114 2024-04-12T21:55:25.235 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:24 smithi097 ceph-mon[24835]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:25.235 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:24 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:55:25.235 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:24 smithi097 ceph-mon[29620]: Deploying daemon node-exporter.b on smithi114 2024-04-12T21:55:25.235 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:24 smithi097 ceph-mon[29620]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:25.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:24 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:55:25.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:24 smithi114 ceph-mon[25759]: Deploying daemon node-exporter.b on smithi114 2024-04-12T21:55:25.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:24 smithi114 ceph-mon[25759]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:25.244 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:25 smithi114 bash[41087]: Getting image source signatures 2024-04-12T21:55:25.244 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:25 smithi114 bash[41087]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-04-12T21:55:25.244 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:25 smithi114 bash[41087]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-04-12T21:55:25.244 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:25 smithi114 bash[41087]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-04-12T21:55:25.786 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-12T21:55:25.786 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":59,"fsid":"80bf7680-f916-11ee-bc8d-c7b262605968","created":"2024-04-12T21:49:48.426985+0000","modified":"2024-04-12T21:55:00.526020+0000","last_up_change":"2024-04-12T21:54:44.911811+0000","last_in_change":"2024-04-12T21:54:28.414803+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-12T21:52:54.377978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"315f320f-95fd-4b18-ab6a-417ba7651667","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":2413859098},{"type":"v1","addr":"172.21.15.97:6803","nonce":2413859098}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":2413859098},{"type":"v1","addr":"172.21.15.97:6805","nonce":2413859098}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":2413859098},{"type":"v1","addr":"172.21.15.97:6809","nonce":2413859098}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":2413859098},{"type":"v1","addr":"172.21.15.97:6807","nonce":2413859098}]},"public_addr":"172.21.15.97:6803/2413859098","cluster_addr":"172.21.15.97:6805/2413859098","heartbeat_back_addr":"172.21.15.97:6809/2413859098","heartbeat_front_addr":"172.21.15.97:6807/2413859098","state":["exists","up"]},{"osd":1,"uuid":"c58dc94b-86d4-4b83-95ad-d3779a5aa46e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":1786985012},{"type":"v1","addr":"172.21.15.97:6811","nonce":1786985012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":1786985012},{"type":"v1","addr":"172.21.15.97:6813","nonce":1786985012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":1786985012},{"type":"v1","addr":"172.21.15.97:6817","nonce":1786985012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":1786985012},{"type":"v1","addr":"172.21.15.97:6815","nonce":1786985012}]},"public_addr":"172.21.15.97:6811/1786985012","cluster_addr":"172.21.15.97:6813/1786985012","heartbeat_back_addr":"172.21.15.97:6817/1786985012","heartbeat_front_addr":"172.21.15.97:6815/1786985012","state":["exists","up"]},{"osd":2,"uuid":"4920ee2a-de4d-4072-92bb-dae616a76152","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":1944472297},{"type":"v1","addr":"172.21.15.97:6819","nonce":1944472297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":1944472297},{"type":"v1","addr":"172.21.15.97:6821","nonce":1944472297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":1944472297},{"type":"v1","addr":"172.21.15.97:6825","nonce":1944472297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":1944472297},{"type":"v1","addr":"172.21.15.97:6823","nonce":1944472297}]},"public_addr":"172.21.15.97:6819/1944472297","cluster_addr":"172.21.15.97:6821/1944472297","heartbeat_back_addr":"172.21.15.97:6825/1944472297","heartbeat_front_addr":"172.21.15.97:6823/1944472297","state":["exists","up"]},{"osd":3,"uuid":"928b3bd7-c0df-443c-9a28-5ceb62ae3160","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":75554646},{"type":"v1","addr":"172.21.15.97:6827","nonce":75554646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":75554646},{"type":"v1","addr":"172.21.15.97:6829","nonce":75554646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":75554646},{"type":"v1","addr":"172.21.15.97:6833","nonce":75554646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":75554646},{"type":"v1","addr":"172.21.15.97:6831","nonce":75554646}]},"public_addr":"172.21.15.97:6827/75554646","cluster_addr":"172.21.15.97:6829/75554646","heartbeat_back_addr":"172.21.15.97:6833/75554646","heartbeat_front_addr":"172.21.15.97:6831/75554646","state":["exists","up"]},{"osd":4,"uuid":"02f49773-bcfc-499c-9174-d9cb5991c789","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":2190104221},{"type":"v1","addr":"172.21.15.114:6801","nonce":2190104221}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":2190104221},{"type":"v1","addr":"172.21.15.114:6803","nonce":2190104221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":2190104221},{"type":"v1","addr":"172.21.15.114:6807","nonce":2190104221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":2190104221},{"type":"v1","addr":"172.21.15.114:6805","nonce":2190104221}]},"public_addr":"172.21.15.114:6801/2190104221","cluster_addr":"172.21.15.114:6803/2190104221","heartbeat_back_addr":"172.21.15.114:6807/2190104221","heartbeat_front_addr":"172.21.15.114:6805/2190104221","state":["exists","up"]},{"osd":5,"uuid":"997405e6-e6f2-4c40-b81c-7848c56c062c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":515183706},{"type":"v1","addr":"172.21.15.114:6809","nonce":515183706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":515183706},{"type":"v1","addr":"172.21.15.114:6811","nonce":515183706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":515183706},{"type":"v1","addr":"172.21.15.114:6815","nonce":515183706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":515183706},{"type":"v1","addr":"172.21.15.114:6813","nonce":515183706}]},"public_addr":"172.21.15.114:6809/515183706","cluster_addr":"172.21.15.114:6811/515183706","heartbeat_back_addr":"172.21.15.114:6815/515183706","heartbeat_front_addr":"172.21.15.114:6813/515183706","state":["exists","up"]},{"osd":6,"uuid":"68b36dad-0aff-40a4-b8f7-8c1dcf42a5c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":2512498411},{"type":"v1","addr":"172.21.15.114:6817","nonce":2512498411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":2512498411},{"type":"v1","addr":"172.21.15.114:6819","nonce":2512498411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":2512498411},{"type":"v1","addr":"172.21.15.114:6823","nonce":2512498411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":2512498411},{"type":"v1","addr":"172.21.15.114:6821","nonce":2512498411}]},"public_addr":"172.21.15.114:6817/2512498411","cluster_addr":"172.21.15.114:6819/2512498411","heartbeat_back_addr":"172.21.15.114:6823/2512498411","heartbeat_front_addr":"172.21.15.114:6821/2512498411","state":["exists","up"]},{"osd":7,"uuid":"719dfe0e-c27b-4ada-a3e0-e381d70eb53b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":1147837664},{"type":"v1","addr":"172.21.15.114:6825","nonce":1147837664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":1147837664},{"type":"v1","addr":"172.21.15.114:6827","nonce":1147837664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":1147837664},{"type":"v1","addr":"172.21.15.114:6831","nonce":1147837664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":1147837664},{"type":"v1","addr":"172.21.15.114:6829","nonce":1147837664}]},"public_addr":"172.21.15.114:6825/1147837664","cluster_addr":"172.21.15.114:6827/1147837664","heartbeat_back_addr":"172.21.15.114:6831/1147837664","heartbeat_front_addr":"172.21.15.114:6829/1147837664","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:52:02.934510+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:52:26.942832+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:52:51.162399+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:53:15.754757+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:53:36.448575+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:53:58.560201+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:54:20.557106+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:54:43.500361+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.97:0/2357886921":"2024-04-13T21:55:00.525973+0000","172.21.15.97:6801/2093806388":"2024-04-13T21:55:00.525973+0000","172.21.15.97:6800/2093806388":"2024-04-13T21:55:00.525973+0000","172.21.15.97:0/2789980025":"2024-04-13T21:55:00.525973+0000","172.21.15.97:0/1481749366":"2024-04-13T21:55:00.525973+0000","172.21.15.97:0/831596468":"2024-04-13T21:50:43.236028+0000","172.21.15.97:0/1510317264":"2024-04-13T21:55:00.525973+0000","172.21.15.97:0/1562080679":"2024-04-13T21:50:43.236028+0000","172.21.15.97:6801/258588252":"2024-04-13T21:50:43.236028+0000","172.21.15.97:0/2696421935":"2024-04-13T21:50:43.236028+0000","172.21.15.97:6800/258588252":"2024-04-13T21:50:43.236028+0000","172.21.15.97:0/2460245404":"2024-04-13T21:50:20.870746+0000","172.21.15.97:0/1828347223":"2024-04-13T21:50:20.870746+0000","172.21.15.97:6800/1796168157":"2024-04-13T21:50:20.870746+0000","172.21.15.97:6801/1796168157":"2024-04-13T21:50:20.870746+0000","172.21.15.97:0/1917553394":"2024-04-13T21:50:20.870746+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-12T21:55:26.147 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:25 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/2530985943' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:55:26.147 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:25 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/2530985943' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:55:26.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:25 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/2530985943' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:55:26.411 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:55:26 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: ::ffff:172.21.15.114 - - [12/Apr/2024:21:55:26] "GET /metrics HTTP/1.1" 200 191521 "" "Prometheus/2.33.4" 2024-04-12T21:55:26.786 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-12T21:55:26.786 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-12T21:55:26.787 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-12T21:55:26.787 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-12T21:55:26.787 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-12T21:55:26.788 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-12T21:55:26.788 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-12T21:55:26.789 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-12T21:55:26.985 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:26 smithi114 bash[41087]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-04-12T21:55:26.985 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:26 smithi114 bash[41087]: Writing manifest to image destination 2024-04-12T21:55:27.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:26 smithi097 ceph-mon[24835]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:27.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:26 smithi097 ceph-mon[29620]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:27.224 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:55:27.230 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:55:27.231 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:55:27.236 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:55:27.236 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:55:27.236 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:55:27.236 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:55:27.238 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:55:27.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:26 smithi114 ceph-mon[25759]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:27.244 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:27 smithi114 podman[41087]: 2024-04-12 21:55:27.124129948 +0000 UTC m=+2.384353917 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-04-12T21:55:27.244 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:27 smithi114 podman[41087]: 2024-04-12 21:55:27.189353543 +0000 UTC m=+2.449577519 container create 20d4da15891ea70628d2dfc691b675971db256218e5c1f1f8f3314c945a078f1 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-12T21:55:27.744 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:27 smithi114 podman[41087]: 2024-04-12 21:55:27.374683521 +0000 UTC m=+2.634907496 container init 20d4da15891ea70628d2dfc691b675971db256218e5c1f1f8f3314c945a078f1 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-12T21:55:27.745 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:27 smithi114 podman[41087]: 2024-04-12 21:55:27.377487037 +0000 UTC m=+2.637711011 container start 20d4da15891ea70628d2dfc691b675971db256218e5c1f1f8f3314c945a078f1 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-12T21:55:27.745 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[41148]: ts=2024-04-12T21:55:27.381Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-04-12T21:55:27.745 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[41148]: ts=2024-04-12T21:55:27.381Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-04-12T21:55:27.745 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[41148]: ts=2024-04-12T21:55:27.381Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-04-12T21:55:27.745 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[41148]: ts=2024-04-12T21:55:27.381Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-04-12T21:55:27.745 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[41148]: ts=2024-04-12T21:55:27.382Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-04-12T21:55:27.745 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[41148]: ts=2024-04-12T21:55:27.382Z caller=node_exporter.go:115 level=info collector=arp 2024-04-12T21:55:27.745 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[41148]: ts=2024-04-12T21:55:27.382Z caller=node_exporter.go:115 level=info collector=bcache 2024-04-12T21:55:27.745 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[41148]: ts=2024-04-12T21:55:27.382Z caller=node_exporter.go:115 level=info collector=bonding 2024-04-12T21:55:27.746 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[41148]: ts=2024-04-12T21:55:27.382Z caller=node_exporter.go:115 level=info collector=btrfs 2024-04-12T21:55:27.746 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[41148]: ts=2024-04-12T21:55:27.382Z caller=node_exporter.go:115 level=info collector=conntrack 2024-04-12T21:55:27.746 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[41148]: ts=2024-04-12T21:55:27.382Z caller=node_exporter.go:115 level=info collector=cpu 2024-04-12T21:55:27.746 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[41148]: ts=2024-04-12T21:55:27.382Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-04-12T21:55:27.746 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[41148]: ts=2024-04-12T21:55:27.382Z caller=node_exporter.go:115 level=info collector=diskstats 2024-04-12T21:55:27.746 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[41148]: ts=2024-04-12T21:55:27.382Z caller=node_exporter.go:115 level=info collector=dmi 2024-04-12T21:55:27.746 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[41148]: ts=2024-04-12T21:55:27.382Z caller=node_exporter.go:115 level=info collector=edac 2024-04-12T21:55:27.746 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[41148]: ts=2024-04-12T21:55:27.382Z caller=node_exporter.go:115 level=info collector=entropy 2024-04-12T21:55:27.746 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[41148]: ts=2024-04-12T21:55:27.382Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-04-12T21:55:27.746 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[41148]: ts=2024-04-12T21:55:27.382Z caller=node_exporter.go:115 level=info collector=filefd 2024-04-12T21:55:27.746 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[41148]: ts=2024-04-12T21:55:27.382Z caller=node_exporter.go:115 level=info collector=filesystem 2024-04-12T21:55:27.746 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[41148]: ts=2024-04-12T21:55:27.382Z caller=node_exporter.go:115 level=info collector=hwmon 2024-04-12T21:55:27.747 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[41148]: ts=2024-04-12T21:55:27.382Z caller=node_exporter.go:115 level=info collector=infiniband 2024-04-12T21:55:27.747 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[41148]: ts=2024-04-12T21:55:27.382Z caller=node_exporter.go:115 level=info collector=ipvs 2024-04-12T21:55:27.747 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[41148]: ts=2024-04-12T21:55:27.382Z caller=node_exporter.go:115 level=info collector=loadavg 2024-04-12T21:55:27.747 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[41148]: ts=2024-04-12T21:55:27.382Z caller=node_exporter.go:115 level=info collector=mdadm 2024-04-12T21:55:27.747 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[41148]: ts=2024-04-12T21:55:27.382Z caller=node_exporter.go:115 level=info collector=meminfo 2024-04-12T21:55:27.747 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[41148]: ts=2024-04-12T21:55:27.382Z caller=node_exporter.go:115 level=info collector=netclass 2024-04-12T21:55:27.747 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[41148]: ts=2024-04-12T21:55:27.382Z caller=node_exporter.go:115 level=info collector=netdev 2024-04-12T21:55:27.747 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[41148]: ts=2024-04-12T21:55:27.382Z caller=node_exporter.go:115 level=info collector=netstat 2024-04-12T21:55:27.747 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[41148]: ts=2024-04-12T21:55:27.382Z caller=node_exporter.go:115 level=info collector=nfs 2024-04-12T21:55:27.747 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[41148]: ts=2024-04-12T21:55:27.382Z caller=node_exporter.go:115 level=info collector=nfsd 2024-04-12T21:55:27.747 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[41148]: ts=2024-04-12T21:55:27.382Z caller=node_exporter.go:115 level=info collector=nvme 2024-04-12T21:55:27.747 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[41148]: ts=2024-04-12T21:55:27.382Z caller=node_exporter.go:115 level=info collector=os 2024-04-12T21:55:27.748 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[41148]: ts=2024-04-12T21:55:27.382Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-04-12T21:55:27.748 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[41148]: ts=2024-04-12T21:55:27.382Z caller=node_exporter.go:115 level=info collector=pressure 2024-04-12T21:55:27.748 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[41148]: ts=2024-04-12T21:55:27.382Z caller=node_exporter.go:115 level=info collector=rapl 2024-04-12T21:55:27.748 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[41148]: ts=2024-04-12T21:55:27.382Z caller=node_exporter.go:115 level=info collector=schedstat 2024-04-12T21:55:27.748 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[41148]: ts=2024-04-12T21:55:27.382Z caller=node_exporter.go:115 level=info collector=sockstat 2024-04-12T21:55:27.748 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[41148]: ts=2024-04-12T21:55:27.382Z caller=node_exporter.go:115 level=info collector=softnet 2024-04-12T21:55:27.748 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[41148]: ts=2024-04-12T21:55:27.382Z caller=node_exporter.go:115 level=info collector=stat 2024-04-12T21:55:27.748 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[41148]: ts=2024-04-12T21:55:27.382Z caller=node_exporter.go:115 level=info collector=tapestats 2024-04-12T21:55:27.748 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[41148]: ts=2024-04-12T21:55:27.382Z caller=node_exporter.go:115 level=info collector=textfile 2024-04-12T21:55:27.748 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[41148]: ts=2024-04-12T21:55:27.382Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-04-12T21:55:27.749 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[41148]: ts=2024-04-12T21:55:27.382Z caller=node_exporter.go:115 level=info collector=time 2024-04-12T21:55:27.749 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[41148]: ts=2024-04-12T21:55:27.382Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-04-12T21:55:27.749 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[41148]: ts=2024-04-12T21:55:27.382Z caller=node_exporter.go:115 level=info collector=uname 2024-04-12T21:55:27.749 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[41148]: ts=2024-04-12T21:55:27.382Z caller=node_exporter.go:115 level=info collector=vmstat 2024-04-12T21:55:27.749 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[41148]: ts=2024-04-12T21:55:27.382Z caller=node_exporter.go:115 level=info collector=xfs 2024-04-12T21:55:27.749 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[41148]: ts=2024-04-12T21:55:27.382Z caller=node_exporter.go:115 level=info collector=zfs 2024-04-12T21:55:27.749 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[41148]: ts=2024-04-12T21:55:27.382Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-04-12T21:55:27.749 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[41148]: ts=2024-04-12T21:55:27.382Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-04-12T21:55:27.749 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:27 smithi114 bash[41087]: 20d4da15891ea70628d2dfc691b675971db256218e5c1f1f8f3314c945a078f1 2024-04-12T21:55:27.749 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 21:55:27 smithi114 systemd[1]: Started Ceph node-exporter.b for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T21:55:28.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:28 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:55:28.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:28 smithi097 ceph-mon[24835]: Deploying daemon alertmanager.a on smithi097 2024-04-12T21:55:28.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:28 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:55:28.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:28 smithi097 ceph-mon[29620]: Deploying daemon alertmanager.a on smithi097 2024-04-12T21:55:28.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:28 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:55:28.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:28 smithi114 ceph-mon[25759]: Deploying daemon alertmanager.a on smithi097 2024-04-12T21:55:29.819 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:29 smithi097 ceph-mon[24835]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:29.819 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:29 smithi097 ceph-mon[29620]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:29.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:29 smithi114 ceph-mon[25759]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:31.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:31 smithi097 ceph-mon[24835]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:31.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:31 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:55:31.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:31 smithi097 ceph-mon[29620]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:31.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:31 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:55:31.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:31 smithi114 ceph-mon[25759]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:31.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:31 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:55:33.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:33 smithi097 ceph-mon[24835]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:33.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:33 smithi097 ceph-mon[29620]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:33.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:33 smithi114 ceph-mon[25759]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:34.744 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:55:34 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: ::ffff:172.21.15.114 - - [12/Apr/2024:21:55:34] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-12T21:55:35.844 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:35 smithi097 ceph-mon[29620]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:35.844 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:35 smithi097 ceph-mon[24835]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:35.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:35 smithi114 ceph-mon[25759]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:36.563 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:55:36 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: ::ffff:172.21.15.114 - - [12/Apr/2024:21:55:36] "GET /metrics HTTP/1.1" 200 191495 "" "Prometheus/2.33.4" 2024-04-12T21:55:37.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:37 smithi114 ceph-mon[25759]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:37.908 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:37 smithi097 ceph-mon[24835]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:37.909 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:37 smithi097 ceph-mon[29620]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:39.608 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:39 smithi097 ceph-mon[24835]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:39.608 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:39 smithi097 ceph-mon[29620]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:39.746 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:39 smithi114 ceph-mon[25759]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:40.573 INFO:teuthology.orchestra.run.smithi097.stdout:163208757269 2024-04-12T21:55:40.574 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-12T21:55:41.718 INFO:teuthology.orchestra.run.smithi097.stdout:77309411364 2024-04-12T21:55:41.718 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-12T21:55:41.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:41 smithi097 ceph-mon[24835]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:41.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:41 smithi097 ceph-mon[29620]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:41.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:41 smithi114 ceph-mon[25759]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:43.068 INFO:teuthology.orchestra.run.smithi097.stdout:107374182432 2024-04-12T21:55:43.068 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-12T21:55:43.492 INFO:teuthology.orchestra.run.smithi097.stdout:128849018904 2024-04-12T21:55:43.492 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-12T21:55:43.871 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:43 smithi097 ceph-mon[24835]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:43.872 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:43 smithi097 ceph-mon[29620]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:43.893 INFO:teuthology.orchestra.run.smithi097.stdout:55834574889 2024-04-12T21:55:43.893 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-12T21:55:43.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:43 smithi114 ceph-mon[25759]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:44.716 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:55:44.741 INFO:teuthology.orchestra.run.smithi097.stdout:197568495634 2024-04-12T21:55:44.741 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-12T21:55:44.744 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:55:44 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: ::ffff:172.21.15.114 - - [12/Apr/2024:21:55:44] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-12T21:55:45.218 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:55:45.219 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:55:45.219 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:55:45.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:45 smithi097 ceph-mon[29620]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:45.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:45 smithi097 ceph-mon[24835]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:45.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:45 smithi114 ceph-mon[25759]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:46.398 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:55:46 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: ::ffff:172.21.15.114 - - [12/Apr/2024:21:55:46] "GET /metrics HTTP/1.1" 200 191494 "" "Prometheus/2.33.4" 2024-04-12T21:55:47.581 INFO:teuthology.orchestra.run.smithi097.stdout:231928233998 2024-04-12T21:55:47.582 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-12T21:55:47.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:47 smithi097 ceph-mon[24835]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:47.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:47 smithi097 ceph-mon[29620]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:47.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:47 smithi114 ceph-mon[25759]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:49.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:49 smithi097 ceph-mon[24835]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:49.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:49 smithi097 ceph-mon[29620]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:49.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:49 smithi114 ceph-mon[25759]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:50.904 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:55:50.905 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:55:51.397 INFO:teuthology.orchestra.run.smithi097.stdout:34359738414 2024-04-12T21:55:51.397 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-12T21:55:51.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:51 smithi097 ceph-mon[24835]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:51.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:51 smithi097 ceph-mon[29620]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:51.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:51 smithi114 ceph-mon[25759]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:52.674 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:55:52.797 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:52 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/3562068003' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-12T21:55:52.797 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:52 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/3562068003' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-12T21:55:52.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:52 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/3562068003' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-12T21:55:53.677 INFO:teuthology.orchestra.run.smithi097.stdout:197568495636 2024-04-12T21:55:53.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:53 smithi097 ceph-mon[24835]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:53.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:53 smithi097 ceph-mon[29620]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:53.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:53 smithi114 ceph-mon[25759]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:54.744 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:55:54 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: ::ffff:172.21.15.114 - - [12/Apr/2024:21:55:54] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-12T21:55:54.804 INFO:teuthology.orchestra.run.smithi097.stdout:55834574892 2024-04-12T21:55:55.246 INFO:teuthology.orchestra.run.smithi097.stdout:77309411367 2024-04-12T21:55:55.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:55 smithi097 ceph-mon[24835]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:55.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:55 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/2915176163' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-12T21:55:55.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:55 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/4098326425' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-12T21:55:55.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:55 smithi097 ceph-mon[29620]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:55.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:55 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/2915176163' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-12T21:55:55.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:55 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/4098326425' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-12T21:55:55.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:55 smithi114 ceph-mon[25759]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:55.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:55 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/2915176163' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-12T21:55:55.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:55 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/4098326425' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-12T21:55:56.238 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:55:56 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: ::ffff:172.21.15.114 - - [12/Apr/2024:21:55:56] "GET /metrics HTTP/1.1" 200 191494 "" "Prometheus/2.33.4" 2024-04-12T21:55:57.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:57 smithi097 ceph-mon[24835]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:57.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:57 smithi097 ceph-mon[29620]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:57.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:57 smithi114 ceph-mon[25759]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:58.276 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574889 got 55834574892 for osd.1 2024-04-12T21:55:58.277 DEBUG:teuthology.parallel:result is None 2024-04-12T21:55:58.661 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 21:55:58 smithi097 podman[46990]: 2024-04-12 21:55:58.475169996 +0000 UTC m=+1.068116003 container create 21406e986e92db067752f3e8ca5737c27900fa8672bc6b3b6c4429fbeb59e865 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-12T21:55:58.955 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 21:55:58 smithi097 podman[46990]: 2024-04-12 21:55:58.862228225 +0000 UTC m=+1.455174231 container init 21406e986e92db067752f3e8ca5737c27900fa8672bc6b3b6c4429fbeb59e865 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-12T21:55:58.956 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 21:55:58 smithi097 podman[46990]: 2024-04-12 21:55:58.86512372 +0000 UTC m=+1.458069733 container start 21406e986e92db067752f3e8ca5737c27900fa8672bc6b3b6c4429fbeb59e865 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-12T21:55:58.956 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 21:55:58 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[47012]: level=info ts=2024-04-12T21:55:58.876Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-04-12T21:55:58.956 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 21:55:58 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[47012]: level=info ts=2024-04-12T21:55:58.876Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-04-12T21:55:58.956 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 21:55:58 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[47012]: level=info ts=2024-04-12T21:55:58.877Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.97 port=9094 2024-04-12T21:55:58.956 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 21:55:58 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[47012]: level=info ts=2024-04-12T21:55:58.879Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-04-12T21:55:58.956 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 21:55:58 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[47012]: level=info ts=2024-04-12T21:55:58.904Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-12T21:55:58.956 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 21:55:58 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[47012]: level=info ts=2024-04-12T21:55:58.904Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-12T21:55:58.956 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 21:55:58 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[47012]: level=info ts=2024-04-12T21:55:58.907Z caller=main.go:518 msg=Listening address=:9093 2024-04-12T21:55:58.956 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 21:55:58 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[47012]: level=info ts=2024-04-12T21:55:58.907Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-04-12T21:55:58.956 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 21:55:58 smithi097 bash[46990]: 21406e986e92db067752f3e8ca5737c27900fa8672bc6b3b6c4429fbeb59e865 2024-04-12T21:55:59.381 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 21:55:58 smithi097 systemd[1]: Started Ceph alertmanager.a for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T21:55:59.384 INFO:teuthology.orchestra.run.smithi097.stdout:231928234001 2024-04-12T21:55:59.657 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:59 smithi114 ceph-mon[25759]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:59.657 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:59 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:55:59.657 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:59 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:55:59.657 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:59 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:55:59.657 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:55:59 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:55:59.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:59 smithi097 ceph-mon[24835]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:59.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:59 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:55:59.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:59 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:55:59.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:59 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:55:59.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:55:59 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:55:59.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:59 smithi097 ceph-mon[29620]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:55:59.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:59 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:55:59.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:59 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:55:59.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:59 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:55:59.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:55:59 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:55:59.909 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495634 got 197568495636 for osd.6 2024-04-12T21:55:59.909 DEBUG:teuthology.parallel:result is None 2024-04-12T21:56:00.626 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:00 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/3522253000' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-12T21:56:00.627 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:00 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/3522253000' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-12T21:56:00.656 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411364 got 77309411367 for osd.2 2024-04-12T21:56:00.656 DEBUG:teuthology.parallel:result is None 2024-04-12T21:56:00.673 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:00 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/3522253000' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-12T21:56:01.161 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 21:56:00 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[47012]: level=info ts=2024-04-12T21:56:00.879Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000832071s 2024-04-12T21:56:01.597 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:56:01.622 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928233998 got 231928234001 for osd.7 2024-04-12T21:56:01.622 DEBUG:teuthology.parallel:result is None 2024-04-12T21:56:01.855 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:01 smithi097 ceph-mon[29620]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:56:01.855 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:01 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:01.855 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:01 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T21:56:01.856 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:01 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T21:56:01.856 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:01 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T21:56:01.856 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:01 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T21:56:01.856 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:01 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:01.856 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:01 smithi097 ceph-mon[24835]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:56:01.856 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:01 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:01.856 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:01 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T21:56:01.856 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:01 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T21:56:01.856 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:01 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T21:56:01.856 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:01 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T21:56:01.857 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:01 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:01.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:01 smithi114 ceph-mon[25759]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:56:01.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:01 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:01.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:01 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T21:56:01.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:01 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T21:56:01.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:01 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T21:56:01.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:01 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T21:56:01.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:01 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:03.211 INFO:teuthology.orchestra.run.smithi097.stdout:107374182436 2024-04-12T21:56:03.223 INFO:teuthology.orchestra.run.smithi097.stdout:128849018910 2024-04-12T21:56:03.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:03 smithi097 ceph-mon[24835]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:56:03.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:03 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/1784360569' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-12T21:56:03.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:03 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/2357093261' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-12T21:56:03.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:03 smithi097 ceph-mon[29620]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:56:03.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:03 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/1784360569' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-12T21:56:03.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:03 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/2357093261' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-12T21:56:03.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:03 smithi114 ceph-mon[25759]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:56:03.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:03 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/1784360569' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-12T21:56:03.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:03 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/2357093261' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-12T21:56:04.744 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:56:04 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: ::ffff:172.21.15.114 - - [12/Apr/2024:21:56:04] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-12T21:56:05.383 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182432 got 107374182436 for osd.3 2024-04-12T21:56:05.383 DEBUG:teuthology.parallel:result is None 2024-04-12T21:56:05.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:05 smithi097 ceph-mon[24835]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:56:05.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:05 smithi097 ceph-mon[29620]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:56:05.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:05 smithi114 ceph-mon[25759]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:56:06.500 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:56:06 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: ::ffff:172.21.15.114 - - [12/Apr/2024:21:56:06] "GET /metrics HTTP/1.1" 200 191488 "" "Prometheus/2.33.4" 2024-04-12T21:56:06.538 INFO:teuthology.orchestra.run.smithi097.stdout:34359738418 2024-04-12T21:56:06.883 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:06 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/329058140' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-12T21:56:06.884 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:06 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/329058140' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-12T21:56:06.908 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018904 got 128849018910 for osd.4 2024-04-12T21:56:06.908 DEBUG:teuthology.parallel:result is None 2024-04-12T21:56:06.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:06 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/329058140' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-12T21:56:07.522 INFO:teuthology.orchestra.run.smithi097.stdout:163208757275 2024-04-12T21:56:07.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:07 smithi097 ceph-mon[24835]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:56:07.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:07 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/129003463' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-12T21:56:07.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:07 smithi097 ceph-mon[29620]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:56:07.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:07 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/129003463' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-12T21:56:07.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:07 smithi114 ceph-mon[25759]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:56:07.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:07 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/129003463' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-12T21:56:08.056 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738414 got 34359738418 for osd.0 2024-04-12T21:56:08.057 DEBUG:teuthology.parallel:result is None 2024-04-12T21:56:08.659 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757269 got 163208757275 for osd.5 2024-04-12T21:56:08.659 DEBUG:teuthology.parallel:result is None 2024-04-12T21:56:08.659 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-12T21:56:08.659 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- ceph pg dump --format=json 2024-04-12T21:56:08.901 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:56:09.161 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 21:56:08 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[47012]: level=info ts=2024-04-12T21:56:08.882Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.003836974s 2024-04-12T21:56:09.519 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:09 smithi097 ceph-mon[24835]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:56:09.519 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:09 smithi097 ceph-mon[29620]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:56:09.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:09 smithi114 ceph-mon[25759]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:56:11.785 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:11 smithi097 ceph-mon[24835]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:56:11.785 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:11 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:11.785 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:11 smithi097 ceph-mon[29620]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:56:11.785 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:11 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:11.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:11 smithi114 ceph-mon[25759]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:56:11.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:11 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:12.204 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-12T21:56:12.206 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-04-12T21:56:12.926 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:12 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:12.927 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:12 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:12.927 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:12 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:12.927 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:12 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-12T21:56:12.927 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:12 smithi097 ceph-mon[24835]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-12T21:56:12.927 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:12 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:12.927 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:12 smithi097 ceph-mon[24835]: Deploying daemon grafana.a on smithi114 2024-04-12T21:56:12.927 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:12 smithi097 ceph-mon[24835]: from='client.24523 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-12T21:56:12.927 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:12 smithi097 ceph-mon[24835]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:56:12.927 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:12 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:12.928 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:12 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:12.928 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:12 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:12.928 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:12 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-12T21:56:12.928 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:12 smithi097 ceph-mon[29620]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-12T21:56:12.928 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:12 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:12.928 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:12 smithi097 ceph-mon[29620]: Deploying daemon grafana.a on smithi114 2024-04-12T21:56:12.928 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:12 smithi097 ceph-mon[29620]: from='client.24523 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-12T21:56:12.928 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:12 smithi097 ceph-mon[29620]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:56:12.929 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2024-04-12T21:56:10.599149+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48840,"kb_used_data":6088,"kb_used_omap":0,"kb_used_meta":42688,"kb_avail":749912376,"statfs":{"total":767960285184,"available":767910273024,"internally_reserved":0,"allocated":6234112,"data_stored":3488444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43712512},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002127"},"pg_stats":[{"pgid":"1.0","version":"59'128","reported_seq":266,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-12T21:55:00.660635+0000","last_change":"2024-04-12T21:54:48.942122+0000","last_active":"2024-04-12T21:55:00.660635+0000","last_peered":"2024-04-12T21:55:00.660635+0000","last_clean":"2024-04-12T21:55:00.660635+0000","last_became_active":"2024-04-12T21:54:48.941565+0000","last_became_peered":"2024-04-12T21:54:48.941565+0000","last_unstale":"2024-04-12T21:55:00.660635+0000","last_undegraded":"2024-04-12T21:55:00.660635+0000","last_fullsized":"2024-04-12T21:55:00.660635+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:52:55.128817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:52:55.128817+0000","last_clean_scrub_stamp":"2024-04-12T21:52:55.128817+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:05:30.781366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234003,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6088,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739064,"statfs":{"total":95995035648,"available":95988801536,"internally_reserved":0,"allocated":1114112,"data_stored":766769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 12 21:55:46 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.66700000000000004}]},{"osd":1,"last update":"Fri Apr 12 21:55:46 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.65000000000000002}]},{"osd":2,"last update":"Fri Apr 12 21:55:46 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.69499999999999995}]},{"osd":3,"last update":"Fri Apr 12 21:55:46 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.72799999999999998}]},{"osd":4,"last update":"Fri Apr 12 21:55:46 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.56399999999999995}]},{"osd":5,"last update":"Fri Apr 12 21:55:46 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.54900000000000004}]},{"osd":6,"last update":"Fri Apr 12 21:55:46 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.628},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.67400000000000004}]}]},{"osd":6,"up_from":46,"seq":197568495640,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6140,"kb_used_data":1076,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739012,"statfs":{"total":95995035648,"available":95988748288,"internally_reserved":0,"allocated":1101824,"data_stored":765966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 12 21:55:23 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.64100000000000001}]},{"osd":1,"last update":"Fri Apr 12 21:55:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.629},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.60199999999999998}]},{"osd":2,"last update":"Fri Apr 12 21:55:23 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.501}]},{"osd":3,"last update":"Fri Apr 12 21:55:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.61599999999999999}]},{"osd":4,"last update":"Fri Apr 12 21:55:23 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.436,"5min":0.436,"15min":0.436},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.47999999999999998}]},{"osd":5,"last update":"Fri Apr 12 21:55:23 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.65900000000000003}]},{"osd":7,"last update":"Fri Apr 12 21:55:47 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.66900000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574895,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6336,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738816,"statfs":{"total":95995035648,"available":95988547584,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 12 21:55:33 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.33500000000000002,"15min":0.311},"min":{"1min":0.252,"5min":0.161,"15min":0.161},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.436,"5min":0.34399999999999997,"15min":0.312},"min":{"1min":0.31,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.31}]},{"osd":2,"last update":"Fri Apr 12 21:56:00 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.36599999999999999,"15min":0.35599999999999998},"min":{"1min":0.26200000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.378},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.39200000000000002,"15min":0.378},"min":{"1min":0.28699999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.35299999999999998}]},{"osd":3,"last update":"Fri Apr 12 21:55:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.436,"15min":0.42999999999999999},"min":{"1min":0.29499999999999998,"5min":0.219,"15min":0.219},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.438,"5min":0.41999999999999998,"15min":0.41699999999999998},"min":{"1min":0.31900000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.36399999999999999}]},{"osd":4,"last update":"Fri Apr 12 21:55:41 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.57099999999999995,"15min":0.56499999999999995},"min":{"1min":0.35699999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.496},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36699999999999999,"5min":0.315,"15min":0.315},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.48299999999999998}]},{"osd":5,"last update":"Fri Apr 12 21:56:04 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.56399999999999995,"15min":0.56000000000000005},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.627,"5min":0.58299999999999996,"15min":0.57499999999999996},"min":{"1min":0.441,"5min":0.433,"15min":0.433},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.51800000000000002}]},{"osd":6,"last update":"Fri Apr 12 21:55:28 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.50900000000000001}]},{"osd":7,"last update":"Fri Apr 12 21:55:46 2024","interfaces":[{"interface":"back","average":{"1min":1.5649999999999999,"5min":1.5649999999999999,"15min":1.5649999999999999},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":18.506,"5min":18.506,"15min":18.506},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.57399999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738419,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6916,"kb_used_data":1084,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738236,"statfs":{"total":95995035648,"available":95987953664,"internally_reserved":0,"allocated":1110016,"data_stored":766504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Apr 12 21:55:40 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.36099999999999999,"15min":0.33700000000000002},"min":{"1min":0.27500000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.36399999999999999,"15min":0.32600000000000001},"min":{"1min":0.30599999999999999,"5min":0.183,"15min":0.183},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.31900000000000001}]},{"osd":2,"last update":"Fri Apr 12 21:55:57 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.39000000000000001,"15min":0.38200000000000001},"min":{"1min":0.24299999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.56100000000000005,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.434,"5min":0.40699999999999997,"15min":0.39000000000000001},"min":{"1min":0.29499999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.34200000000000003}]},{"osd":3,"last update":"Fri Apr 12 21:55:20 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.45900000000000002,"15min":0.45500000000000002},"min":{"1min":0.29699999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.47299999999999998,"15min":0.46100000000000002},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.41699999999999998}]},{"osd":4,"last update":"Fri Apr 12 21:55:45 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.59499999999999997,"15min":0.58699999999999997},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.57199999999999995,"15min":0.56699999999999995},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.45400000000000001}]},{"osd":5,"last update":"Fri Apr 12 21:56:04 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.59199999999999997,"15min":0.58399999999999996},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.502},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.64100000000000001,"15min":0.63800000000000001},"min":{"1min":0.46000000000000002,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.439}]},{"osd":6,"last update":"Fri Apr 12 21:55:24 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.42799999999999999}]},{"osd":7,"last update":"Fri Apr 12 21:55:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.46200000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6272,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738880,"statfs":{"total":95995035648,"available":95988613120,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 12 21:55:59 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.375,"15min":0.36099999999999999},"min":{"1min":0.255,"5min":0.22,"15min":0.22},"max":{"1min":0.622,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.39300000000000002,"15min":0.38100000000000001},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.33600000000000002}]},{"osd":1,"last update":"Fri Apr 12 21:55:59 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.377,"15min":0.373},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.41499999999999998,"15min":0.40300000000000002},"min":{"1min":0.29699999999999999,"5min":0.251,"15min":0.251},"max":{"1min":0.67800000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.26700000000000002}]},{"osd":3,"last update":"Fri Apr 12 21:55:22 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.46000000000000002,"15min":0.46600000000000003},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.59999999999999998,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.432,"5min":0.45300000000000001,"15min":0.45700000000000002},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.32500000000000001}]},{"osd":4,"last update":"Fri Apr 12 21:55:43 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55200000000000005,"15min":0.55100000000000005},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.58399999999999996,"15min":0.58799999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.79000000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.55700000000000005}]},{"osd":5,"last update":"Fri Apr 12 21:56:03 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.59499999999999997,"15min":0.59099999999999997},"min":{"1min":0.42299999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.83599999999999997,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.57899999999999996,"15min":0.57399999999999995},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.81399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.47099999999999997}]},{"osd":6,"last update":"Fri Apr 12 21:55:22 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.54000000000000004}]},{"osd":7,"last update":"Fri Apr 12 21:55:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.58899999999999997}]}]},{"osd":3,"up_from":25,"seq":107374182437,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5696,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739456,"statfs":{"total":95995035648,"available":95989202944,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 12 21:55:22 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.39000000000000001,"15min":0.39200000000000002},"min":{"1min":0.27000000000000002,"5min":0.248,"15min":0.248},"max":{"1min":0.58499999999999996,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.375},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.36399999999999999}]},{"osd":1,"last update":"Fri Apr 12 21:55:22 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.42999999999999999,"15min":0.434},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.80300000000000005,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.29299999999999998}]},{"osd":2,"last update":"Fri Apr 12 21:55:22 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.44700000000000001,"15min":0.45100000000000001},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.76600000000000001,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.44400000000000001,"15min":0.44800000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.67600000000000005,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.40400000000000003}]},{"osd":4,"last update":"Fri Apr 12 21:55:42 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.56299999999999994,"15min":0.56100000000000005},"min":{"1min":0.42699999999999999,"5min":0.38,"15min":0.38},"max":{"1min":0.80500000000000005,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.55400000000000005,"15min":0.55600000000000005},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.59499999999999997}]},{"osd":5,"last update":"Fri Apr 12 21:55:03 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.75600000000000001}]},{"osd":6,"last update":"Fri Apr 12 21:55:25 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.67300000000000004}]},{"osd":7,"last update":"Fri Apr 12 21:55:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.70699999999999996}]}]},{"osd":4,"up_from":30,"seq":128849018911,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5760,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739392,"statfs":{"total":95995035648,"available":95989137408,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 12 21:55:39 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.56499999999999995,"15min":0.57499999999999996},"min":{"1min":0.34799999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.70299999999999996,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.495,"5min":0.53300000000000003,"15min":0.53900000000000003},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.71499999999999997,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.47699999999999998}]},{"osd":1,"last update":"Fri Apr 12 21:55:39 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.55400000000000005,"15min":0.56499999999999995},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.64300000000000002,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.53000000000000003,"15min":0.53800000000000003},"min":{"1min":0.35199999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.63400000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.42099999999999999}]},{"osd":2,"last update":"Fri Apr 12 21:55:39 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.51800000000000002,"15min":0.52200000000000002},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.67500000000000004,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.56799999999999995,"15min":0.57399999999999995},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.69499999999999995,"5min":0.878,"15min":0.878},"last":0.51000000000000001}]},{"osd":3,"last update":"Fri Apr 12 21:55:39 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.56599999999999995,"15min":0.56799999999999995},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.72599999999999998,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.55200000000000005,"15min":0.55600000000000005},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.67900000000000005,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.49099999999999999}]},{"osd":5,"last update":"Fri Apr 12 21:56:02 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.40500000000000003,"15min":0.40000000000000002},"min":{"1min":0.32200000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.45100000000000001,"15min":0.44600000000000001},"min":{"1min":0.32200000000000001,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.435}]},{"osd":6,"last update":"Fri Apr 12 21:55:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.376}]},{"osd":7,"last update":"Fri Apr 12 21:55:47 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.52000000000000002}]}]},{"osd":5,"up_from":38,"seq":163208757276,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5632,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739520,"statfs":{"total":95995035648,"available":95989268480,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 12 21:56:04 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.93799999999999994,"15min":1.0069999999999999},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.754,"5min":11.653,"15min":11.653},"last":0.754},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.93500000000000005,"15min":0.998},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.70599999999999996,"5min":11.678000000000001,"15min":11.678000000000001},"last":0.68400000000000005}]},{"osd":1,"last update":"Fri Apr 12 21:56:04 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.51000000000000001,"15min":0.50600000000000001},"min":{"1min":0.40899999999999997,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.67000000000000004,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.51100000000000001,"15min":0.502},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.72299999999999998}]},{"osd":2,"last update":"Fri Apr 12 21:56:04 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.93899999999999995,"15min":1.0009999999999999},"min":{"1min":0.44600000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.75900000000000001,"5min":11.663,"15min":11.663},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.505,"15min":0.497},"min":{"1min":0.38200000000000001,"5min":0.314,"15min":0.314},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.80300000000000005}]},{"osd":3,"last update":"Fri Apr 12 21:56:04 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.95899999999999996,"15min":1.0229999999999999},"min":{"1min":0.40000000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.77200000000000002,"5min":11.701000000000001,"15min":11.701000000000001},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.98099999999999998,"15min":1.05},"min":{"1min":0.38200000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.71699999999999997,"5min":11.694000000000001,"15min":11.694000000000001},"last":0.71199999999999997}]},{"osd":4,"last update":"Fri Apr 12 21:56:04 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.40000000000000002,"15min":0.38900000000000001},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.40899999999999997,"15min":0.39700000000000002},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.73999999999999999}]},{"osd":6,"last update":"Fri Apr 12 21:55:22 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.81899999999999995}]},{"osd":7,"last update":"Fri Apr 12 21:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.84799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-12T21:56:12.931 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- ceph pg dump --format=json 2024-04-12T21:56:13.145 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:56:13.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:12 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:13.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:12 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:13.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:12 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:13.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:12 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-12T21:56:13.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:12 smithi114 ceph-mon[25759]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-12T21:56:13.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:12 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:13.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:12 smithi114 ceph-mon[25759]: Deploying daemon grafana.a on smithi114 2024-04-12T21:56:13.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:12 smithi114 ceph-mon[25759]: from='client.24523 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-12T21:56:13.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:12 smithi114 ceph-mon[25759]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:56:14.744 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:56:14 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: ::ffff:172.21.15.114 - - [12/Apr/2024:21:56:14] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-12T21:56:15.023 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-12T21:56:15.024 INFO:teuthology.orchestra.run.smithi097.stderr:dumped all 2024-04-12T21:56:15.692 INFO:teuthology.orchestra.run.smithi097.stdout:{"pg_ready":true,"pg_map":{"version":40,"stamp":"2024-04-12T21:56:14.599741+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48840,"kb_used_data":6088,"kb_used_omap":0,"kb_used_meta":42688,"kb_avail":749912376,"statfs":{"total":767960285184,"available":767910273024,"internally_reserved":0,"allocated":6234112,"data_stored":3488444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43712512},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002070"},"pg_stats":[{"pgid":"1.0","version":"59'128","reported_seq":266,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-12T21:55:00.660635+0000","last_change":"2024-04-12T21:54:48.942122+0000","last_active":"2024-04-12T21:55:00.660635+0000","last_peered":"2024-04-12T21:55:00.660635+0000","last_clean":"2024-04-12T21:55:00.660635+0000","last_became_active":"2024-04-12T21:54:48.941565+0000","last_became_peered":"2024-04-12T21:54:48.941565+0000","last_unstale":"2024-04-12T21:55:00.660635+0000","last_undegraded":"2024-04-12T21:55:00.660635+0000","last_fullsized":"2024-04-12T21:55:00.660635+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:52:55.128817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:52:55.128817+0000","last_clean_scrub_stamp":"2024-04-12T21:52:55.128817+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:05:30.781366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234004,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6088,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739064,"statfs":{"total":95995035648,"available":95988801536,"internally_reserved":0,"allocated":1114112,"data_stored":766769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 12 21:55:46 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.625}]},{"osd":1,"last update":"Fri Apr 12 21:55:46 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.55800000000000005}]},{"osd":2,"last update":"Fri Apr 12 21:55:46 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.497}]},{"osd":3,"last update":"Fri Apr 12 21:55:46 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.61099999999999999}]},{"osd":4,"last update":"Fri Apr 12 21:55:46 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.505},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.42199999999999999}]},{"osd":5,"last update":"Fri Apr 12 21:55:46 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.59299999999999997}]},{"osd":6,"last update":"Fri Apr 12 21:55:46 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.58599999999999997}]}]},{"osd":6,"up_from":46,"seq":197568495641,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6140,"kb_used_data":1076,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739012,"statfs":{"total":95995035648,"available":95988748288,"internally_reserved":0,"allocated":1101824,"data_stored":765966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 12 21:55:23 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.47799999999999998}]},{"osd":1,"last update":"Fri Apr 12 21:55:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.437}]},{"osd":2,"last update":"Fri Apr 12 21:55:23 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.495},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.505}]},{"osd":3,"last update":"Fri Apr 12 21:55:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.56999999999999995}]},{"osd":4,"last update":"Fri Apr 12 21:55:23 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.436,"5min":0.436,"15min":0.436},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.624}]},{"osd":5,"last update":"Fri Apr 12 21:55:23 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.55700000000000005}]},{"osd":7,"last update":"Fri Apr 12 21:55:47 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.64500000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574896,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6336,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738816,"statfs":{"total":95995035648,"available":95988547584,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 12 21:55:33 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.33500000000000002,"15min":0.311},"min":{"1min":0.252,"5min":0.161,"15min":0.161},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.436,"5min":0.34399999999999997,"15min":0.312},"min":{"1min":0.31,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.30399999999999999}]},{"osd":2,"last update":"Fri Apr 12 21:56:00 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.36599999999999999,"15min":0.35599999999999998},"min":{"1min":0.26200000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.39200000000000002,"15min":0.378},"min":{"1min":0.28699999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.33800000000000002}]},{"osd":3,"last update":"Fri Apr 12 21:55:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.436,"15min":0.42999999999999999},"min":{"1min":0.29499999999999998,"5min":0.219,"15min":0.219},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.438,"5min":0.41999999999999998,"15min":0.41699999999999998},"min":{"1min":0.31900000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.35899999999999999}]},{"osd":4,"last update":"Fri Apr 12 21:55:41 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.57099999999999995,"15min":0.56499999999999995},"min":{"1min":0.35699999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36699999999999999,"5min":0.315,"15min":0.315},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.51600000000000001}]},{"osd":5,"last update":"Fri Apr 12 21:56:04 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.56399999999999995,"15min":0.56000000000000005},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.627,"5min":0.58299999999999996,"15min":0.57499999999999996},"min":{"1min":0.441,"5min":0.433,"15min":0.433},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.56699999999999995}]},{"osd":6,"last update":"Fri Apr 12 21:55:28 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.438},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.49199999999999999}]},{"osd":7,"last update":"Fri Apr 12 21:55:46 2024","interfaces":[{"interface":"back","average":{"1min":1.5649999999999999,"5min":1.5649999999999999,"15min":1.5649999999999999},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":18.506,"5min":18.506,"15min":18.506},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.45700000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738420,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6916,"kb_used_data":1084,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738236,"statfs":{"total":95995035648,"available":95987953664,"internally_reserved":0,"allocated":1110016,"data_stored":766504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Apr 12 21:55:40 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.36099999999999999,"15min":0.33700000000000002},"min":{"1min":0.27500000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.36399999999999999,"15min":0.32600000000000001},"min":{"1min":0.30599999999999999,"5min":0.183,"15min":0.183},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.374}]},{"osd":2,"last update":"Fri Apr 12 21:55:57 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.39000000000000001,"15min":0.38200000000000001},"min":{"1min":0.24299999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.56100000000000005,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.434,"5min":0.40699999999999997,"15min":0.39000000000000001},"min":{"1min":0.29499999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.35499999999999998}]},{"osd":3,"last update":"Fri Apr 12 21:55:20 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.45900000000000002,"15min":0.45500000000000002},"min":{"1min":0.29699999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.47299999999999998,"15min":0.46100000000000002},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.39200000000000002}]},{"osd":4,"last update":"Fri Apr 12 21:55:45 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.59499999999999997,"15min":0.58699999999999997},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.57199999999999995,"15min":0.56699999999999995},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.63900000000000001}]},{"osd":5,"last update":"Fri Apr 12 21:56:04 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.59199999999999997,"15min":0.58399999999999996},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.64100000000000001,"15min":0.63800000000000001},"min":{"1min":0.46000000000000002,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.68500000000000005}]},{"osd":6,"last update":"Fri Apr 12 21:55:24 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.59999999999999998}]},{"osd":7,"last update":"Fri Apr 12 21:55:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.63200000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6272,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738880,"statfs":{"total":95995035648,"available":95988613120,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 12 21:55:59 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.375,"15min":0.36099999999999999},"min":{"1min":0.255,"5min":0.22,"15min":0.22},"max":{"1min":0.622,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.432},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.39300000000000002,"15min":0.38100000000000001},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.495}]},{"osd":1,"last update":"Fri Apr 12 21:55:59 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.377,"15min":0.373},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.41499999999999998,"15min":0.40300000000000002},"min":{"1min":0.29699999999999999,"5min":0.251,"15min":0.251},"max":{"1min":0.67800000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.46500000000000002}]},{"osd":3,"last update":"Fri Apr 12 21:55:22 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.46000000000000002,"15min":0.46600000000000003},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.59999999999999998,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.432,"5min":0.45300000000000001,"15min":0.45700000000000002},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.51400000000000001}]},{"osd":4,"last update":"Fri Apr 12 21:55:43 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55200000000000005,"15min":0.55100000000000005},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.58399999999999996,"15min":0.58799999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.79000000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.65300000000000002}]},{"osd":5,"last update":"Fri Apr 12 21:56:03 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.59499999999999997,"15min":0.59099999999999997},"min":{"1min":0.42299999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.83599999999999997,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.57899999999999996,"15min":0.57399999999999995},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.81399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.82299999999999995}]},{"osd":6,"last update":"Fri Apr 12 21:55:22 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.70799999999999996}]},{"osd":7,"last update":"Fri Apr 12 21:55:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.72399999999999998}]}]},{"osd":3,"up_from":25,"seq":107374182438,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5696,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739456,"statfs":{"total":95995035648,"available":95989202944,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 12 21:55:22 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.39000000000000001,"15min":0.39200000000000002},"min":{"1min":0.27000000000000002,"5min":0.248,"15min":0.248},"max":{"1min":0.58499999999999996,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.23300000000000001},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.36799999999999999}]},{"osd":1,"last update":"Fri Apr 12 21:55:22 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.316},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.42999999999999999,"15min":0.434},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.80300000000000005,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.34699999999999998}]},{"osd":2,"last update":"Fri Apr 12 21:55:22 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.44700000000000001,"15min":0.45100000000000001},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.76600000000000001,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.38},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.44400000000000001,"15min":0.44800000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.67600000000000005,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.29199999999999998}]},{"osd":4,"last update":"Fri Apr 12 21:55:42 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.56299999999999994,"15min":0.56100000000000005},"min":{"1min":0.42699999999999999,"5min":0.38,"15min":0.38},"max":{"1min":0.80500000000000005,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.55400000000000005,"15min":0.55600000000000005},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.61799999999999999}]},{"osd":5,"last update":"Fri Apr 12 21:56:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.69199999999999995,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57999999999999996,"15min":0.58099999999999996},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.75800000000000001,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.66900000000000004}]},{"osd":6,"last update":"Fri Apr 12 21:55:25 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.67900000000000005}]},{"osd":7,"last update":"Fri Apr 12 21:55:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.60499999999999998}]}]},{"osd":4,"up_from":30,"seq":128849018912,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5760,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739392,"statfs":{"total":95995035648,"available":95989137408,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 12 21:55:39 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.56499999999999995,"15min":0.57499999999999996},"min":{"1min":0.34799999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.70299999999999996,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.495,"5min":0.53300000000000003,"15min":0.53900000000000003},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.71499999999999997,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.40100000000000002}]},{"osd":1,"last update":"Fri Apr 12 21:55:39 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.55400000000000005,"15min":0.56499999999999995},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.64300000000000002,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.53000000000000003,"15min":0.53800000000000003},"min":{"1min":0.35199999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.63400000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.50600000000000001}]},{"osd":2,"last update":"Fri Apr 12 21:55:39 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.51800000000000002,"15min":0.52200000000000002},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.67500000000000004,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.56799999999999995,"15min":0.57399999999999995},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.69499999999999995,"5min":0.878,"15min":0.878},"last":0.53200000000000003}]},{"osd":3,"last update":"Fri Apr 12 21:55:39 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.56599999999999995,"15min":0.56799999999999995},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.72599999999999998,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.55200000000000005,"15min":0.55600000000000005},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.67900000000000005,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.54800000000000004}]},{"osd":5,"last update":"Fri Apr 12 21:56:02 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.40500000000000003,"15min":0.40000000000000002},"min":{"1min":0.32200000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.45100000000000001,"15min":0.44600000000000001},"min":{"1min":0.32200000000000001,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.60999999999999999}]},{"osd":6,"last update":"Fri Apr 12 21:55:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.56499999999999995}]},{"osd":7,"last update":"Fri Apr 12 21:55:47 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.58699999999999997}]}]},{"osd":5,"up_from":38,"seq":163208757277,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5632,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739520,"statfs":{"total":95995035648,"available":95989268480,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 12 21:56:04 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.93799999999999994,"15min":1.0069999999999999},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.754,"5min":11.653,"15min":11.653},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.93500000000000005,"15min":0.998},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.70599999999999996,"5min":11.678000000000001,"15min":11.678000000000001},"last":0.73699999999999999}]},{"osd":1,"last update":"Fri Apr 12 21:56:04 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.51000000000000001,"15min":0.50600000000000001},"min":{"1min":0.40899999999999997,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.67000000000000004,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.75},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.51100000000000001,"15min":0.502},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.65600000000000003}]},{"osd":2,"last update":"Fri Apr 12 21:56:04 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.93899999999999995,"15min":1.0009999999999999},"min":{"1min":0.44600000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.75900000000000001,"5min":11.663,"15min":11.663},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.505,"15min":0.497},"min":{"1min":0.38200000000000001,"5min":0.314,"15min":0.314},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.622}]},{"osd":3,"last update":"Fri Apr 12 21:56:04 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.95899999999999996,"15min":1.0229999999999999},"min":{"1min":0.40000000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.77200000000000002,"5min":11.701000000000001,"15min":11.701000000000001},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.98099999999999998,"15min":1.05},"min":{"1min":0.38200000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.71699999999999997,"5min":11.694000000000001,"15min":11.694000000000001},"last":0.71299999999999997}]},{"osd":4,"last update":"Fri Apr 12 21:56:04 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.40000000000000002,"15min":0.38900000000000001},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.40899999999999997,"15min":0.39700000000000002},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.63400000000000001}]},{"osd":6,"last update":"Fri Apr 12 21:55:22 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.55100000000000005}]},{"osd":7,"last update":"Fri Apr 12 21:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.76200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-12T21:56:15.693 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-12T21:56:15.694 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-12T21:56:15.694 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-12T21:56:15.694 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- ceph health --format=json 2024-04-12T21:56:15.902 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:56:15.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:15 smithi114 ceph-mon[25759]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:56:16.024 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:15 smithi097 ceph-mon[24835]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:56:16.024 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:15 smithi097 ceph-mon[29620]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:56:16.332 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:56:16 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: ::ffff:172.21.15.114 - - [12/Apr/2024:21:56:16] "GET /metrics HTTP/1.1" 200 191499 "" "Prometheus/2.33.4" 2024-04-12T21:56:16.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:16 smithi114 ceph-mon[25759]: from='client.14625 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-12T21:56:17.011 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:16 smithi097 ceph-mon[24835]: from='client.14625 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-12T21:56:17.011 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:16 smithi097 ceph-mon[29620]: from='client.14625 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-12T21:56:17.776 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-04-12T21:56:17.776 INFO:teuthology.orchestra.run.smithi097.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-12T21:56:17.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:17 smithi114 ceph-mon[25759]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:56:18.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:17 smithi097 ceph-mon[24835]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:56:18.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:17 smithi097 ceph-mon[29620]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:56:18.365 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-12T21:56:18.366 INFO:tasks.cephadm:Setup complete, yielding 2024-04-12T21:56:18.366 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-12T21:56:18.376 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi097.front.sepia.ceph.com 2024-04-12T21:56:18.377 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest false --force' 2024-04-12T21:56:18.924 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:18 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/4244493833' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-12T21:56:18.925 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:18 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/4244493833' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-12T21:56:18.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:18 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/4244493833' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-12T21:56:19.937 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:19 smithi097 ceph-mon[24835]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:56:19.937 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:19 smithi097 ceph-mon[29620]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:56:19.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:19 smithi114 ceph-mon[25759]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:56:20.991 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-12T21:56:21.003 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi097.front.sepia.ceph.com 2024-04-12T21:56:21.003 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -e sha1=b277feb2c7beb4597d8ecc39d58d5c36b81f613a -- bash -c 'radosgw-admin realm create --rgw-realm=r --default' 2024-04-12T21:56:21.455 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:21 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/2460857125' entity='client.admin' 2024-04-12T21:56:21.455 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:21 smithi114 ceph-mon[25759]: pgmap v43: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:56:21.628 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:21 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/2460857125' entity='client.admin' 2024-04-12T21:56:21.628 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:21 smithi097 ceph-mon[24835]: pgmap v43: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:56:21.628 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:21 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/2460857125' entity='client.admin' 2024-04-12T21:56:21.628 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:21 smithi097 ceph-mon[29620]: pgmap v43: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:56:23.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:23 smithi097 ceph-mon[24835]: pgmap v44: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:56:23.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:23 smithi097 ceph-mon[29620]: pgmap v44: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:56:23.923 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:23 smithi114 ceph-mon[25759]: pgmap v44: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:56:24.244 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 systemd[1]: Starting Ceph grafana.a for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T21:56:24.567 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: ::ffff:172.21.15.114 - - [12/Apr/2024:21:56:24] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-12T21:56:24.568 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 podman[41830]: 2024-04-12 21:56:24.209650023 +0000 UTC m=+0.017235282 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-04-12T21:56:24.568 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 podman[41830]: 2024-04-12 21:56:24.333474596 +0000 UTC m=+0.141059855 container create daa3c792125863c0eee3c7c3c14b153a5626c3648a39cfecf09f2d3ef1900481 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.24.2, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.tags=base rhel8, build-date=2022-03-28T10:36:18.413762, name=ubi8, version=8.5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, description=Ceph Grafana Container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, distribution-scope=public, release=236.1648460182, vendor=Red Hat, Inc., com.redhat.component=ubi8-container, maintainer=Paul Cuzner , io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64) 2024-04-12T21:56:24.568 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 podman[41830]: 2024-04-12 21:56:24.535589758 +0000 UTC m=+0.343175017 container init daa3c792125863c0eee3c7c3c14b153a5626c3648a39cfecf09f2d3ef1900481 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.24.2, io.openshift.tags=base rhel8, build-date=2022-03-28T10:36:18.413762, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=ubi8-container, description=Ceph Grafana Container, summary=Grafana Container configured for Ceph mgr/dashboard integration, name=ubi8, distribution-scope=public, version=8.5, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, maintainer=Paul Cuzner , architecture=x86_64, vendor=Red Hat, Inc., release=236.1648460182, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-12T21:56:24.568 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 podman[41830]: 2024-04-12 21:56:24.53824256 +0000 UTC m=+0.345827828 container start daa3c792125863c0eee3c7c3c14b153a5626c3648a39cfecf09f2d3ef1900481 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container, io.openshift.tags=base rhel8, io.buildah.version=1.24.2, com.redhat.component=ubi8-container, distribution-scope=public, architecture=x86_64, vendor=Red Hat, Inc., name=ubi8, version=8.5, maintainer=Paul Cuzner , io.k8s.display-name=Red Hat Universal Base Image 8, build-date=2022-03-28T10:36:18.413762, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, release=236.1648460182) 2024-04-12T21:56:24.568 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="The state of unified alerting is still not defined. The decision will be made during as we run the database migrations" logger=settings 2024-04-12T21:56:24.568 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=warn msg="falling back to legacy setting of 'min_interval_seconds'; please use the configuration option in the `unified_alerting` section if Grafana 8 alerts are enabled." logger=settings 2024-04-12T21:56:24.568 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2024-04-12T21:56:24.569 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2024-04-12T21:56:24.569 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2024-04-12T21:56:24.569 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2024-04-12T21:56:24.569 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-04-12T21:56:24.569 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-04-12T21:56:24.569 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2024-04-12T21:56:24.569 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2024-04-12T21:56:24.569 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2024-04-12T21:56:24.569 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2024-04-12T21:56:24.569 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2024-04-12T21:56:24.569 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="App mode production" logger=settings 2024-04-12T21:56:24.569 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2024-04-12T21:56:24.569 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=warn msg="SQLite database file has broader permissions than it should" logger=sqlstore path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-04-12T21:56:24.569 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Starting DB migrations" logger=migrator 2024-04-12T21:56:24.569 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create migration_log table" 2024-04-12T21:56:24.720 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-12T21:56:24.720 INFO:teuthology.orchestra.run.smithi097.stdout: "id": "8e06d417-cc67-4293-b394-c4ab3e41b979", 2024-04-12T21:56:24.721 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "r", 2024-04-12T21:56:24.721 INFO:teuthology.orchestra.run.smithi097.stdout: "current_period": "ece10a89-1cd6-493b-b580-36ca0603baa2", 2024-04-12T21:56:24.721 INFO:teuthology.orchestra.run.smithi097.stdout: "epoch": 1 2024-04-12T21:56:24.721 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-12T21:56:24.818 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-mon[25759]: osdmap e60: 8 total, 8 up, 8 in 2024-04-12T21:56:24.818 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/2184930125' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-12T21:56:24.819 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-mon[25759]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-12T21:56:24.819 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create user table" 2024-04-12T21:56:24.819 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.login" 2024-04-12T21:56:24.819 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.email" 2024-04-12T21:56:24.819 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_login - v1" 2024-04-12T21:56:24.819 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_email - v1" 2024-04-12T21:56:24.819 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table user to user_v1 - v1" 2024-04-12T21:56:24.819 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create user table v2" 2024-04-12T21:56:24.819 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_login - v2" 2024-04-12T21:56:24.819 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_email - v2" 2024-04-12T21:56:24.820 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-04-12T21:56:24.820 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table user_v1" 2024-04-12T21:56:24.820 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add column help_flags1 to user table" 2024-04-12T21:56:24.820 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Update user table charset" 2024-04-12T21:56:24.820 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add last_seen_at column to user" 2024-04-12T21:56:24.820 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add missing user data" 2024-04-12T21:56:24.820 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_disabled column to user" 2024-04-12T21:56:24.820 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add index user.login/user.email" 2024-04-12T21:56:24.820 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_service_account column to user" 2024-04-12T21:56:24.820 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create temp user table v1-7" 2024-04-12T21:56:24.820 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v1-7" 2024-04-12T21:56:24.821 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v1-7" 2024-04-12T21:56:24.822 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v1-7" 2024-04-12T21:56:24.822 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v1-7" 2024-04-12T21:56:24.822 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Update temp_user table charset" 2024-04-12T21:56:24.822 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_email - v1" 2024-04-12T21:56:24.822 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_org_id - v1" 2024-04-12T21:56:24.822 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_code - v1" 2024-04-12T21:56:24.822 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_status - v1" 2024-04-12T21:56:24.822 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-04-12T21:56:24.822 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create temp_user v2" 2024-04-12T21:56:24.822 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v2" 2024-04-12T21:56:24.822 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v2" 2024-04-12T21:56:24.822 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v2" 2024-04-12T21:56:24.823 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v2" 2024-04-12T21:56:24.823 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="copy temp_user v1 to v2" 2024-04-12T21:56:24.823 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop temp_user_tmp_qwerty" 2024-04-12T21:56:24.823 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Set created for temp users that will otherwise prematurely expire" 2024-04-12T21:56:24.823 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create star table" 2024-04-12T21:56:24.823 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index star.user_id_dashboard_id" 2024-04-12T21:56:24.823 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create org table v1" 2024-04-12T21:56:24.823 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_name - v1" 2024-04-12T21:56:24.823 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create org_user table v1" 2024-04-12T21:56:24.823 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_org_id - v1" 2024-04-12T21:56:24.823 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_user_org_id_user_id - v1" 2024-04-12T21:56:24.823 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_user_id - v1" 2024-04-12T21:56:24.823 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Update org table charset" 2024-04-12T21:56:24.823 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Update org_user table charset" 2024-04-12T21:56:24.824 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate all Read Only Viewers to Viewers" 2024-04-12T21:56:24.824 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard table" 2024-04-12T21:56:24.824 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard.account_id" 2024-04-12T21:56:24.824 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_account_id_slug" 2024-04-12T21:56:24.824 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_tag table" 2024-04-12T21:56:24.824 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_tag.dasboard_id_term" 2024-04-12T21:56:24.824 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-04-12T21:56:24.824 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard to dashboard_v1 - v1" 2024-04-12T21:56:24.824 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard v2" 2024-04-12T21:56:24.824 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_org_id - v2" 2024-04-12T21:56:24.824 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_org_id_slug - v2" 2024-04-12T21:56:24.824 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard v1 to v2" 2024-04-12T21:56:24.824 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_v1" 2024-04-12T21:56:24.824 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard.data to mediumtext v1" 2024-04-12T21:56:24.825 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add column updated_by in dashboard - v2" 2024-04-12T21:56:24.825 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add column created_by in dashboard - v2" 2024-04-12T21:56:24.825 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add column gnetId in dashboard" 2024-04-12T21:56:24.825 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for gnetId in dashboard" 2024-04-12T21:56:24.825 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_id in dashboard" 2024-04-12T21:56:24.825 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for plugin_id in dashboard" 2024-04-12T21:56:24.825 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_id in dashboard_tag" 2024-04-12T21:56:24.825 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard table charset" 2024-04-12T21:56:24.825 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_tag table charset" 2024-04-12T21:56:24.825 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add column folder_id in dashboard" 2024-04-12T21:56:24.825 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add column isFolder in dashboard" 2024-04-12T21:56:24.825 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add column has_acl in dashboard" 2024-04-12T21:56:24.825 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in dashboard" 2024-04-12T21:56:24.825 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in dashboard" 2024-04-12T21:56:24.826 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index dashboard_org_id_uid" 2024-04-12T21:56:24.826 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_slug" 2024-04-12T21:56:24.826 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard title length" 2024-04-12T21:56:24.826 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index for dashboard_org_id_title_folder_id" 2024-04-12T21:56:24.826 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning" 2024-04-12T21:56:24.826 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-04-12T21:56:24.826 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 bash[41830]: daa3c792125863c0eee3c7c3c14b153a5626c3648a39cfecf09f2d3ef1900481 2024-04-12T21:56:24.826 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning v2" 2024-04-12T21:56:24.826 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 systemd[1]: Started Ceph grafana.a for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T21:56:24.826 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-04-12T21:56:24.826 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-04-12T21:56:24.826 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard_provisioning v1 to v2" 2024-04-12T21:56:24.826 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop dashboard_provisioning_tmp_qwerty" 2024-04-12T21:56:24.826 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add check_sum column" 2024-04-12T21:56:24.827 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_title" 2024-04-12T21:56:24.827 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="delete tags for deleted dashboards" 2024-04-12T21:56:24.827 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="delete stars for deleted dashboards" 2024-04-12T21:56:24.827 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_is_folder" 2024-04-12T21:56:24.827 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table" 2024-04-12T21:56:24.827 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index data_source.account_id" 2024-04-12T21:56:24.827 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index data_source.account_id_name" 2024-04-12T21:56:24.827 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_data_source_account_id - v1" 2024-04-12T21:56:24.827 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_data_source_account_id_name - v1" 2024-04-12T21:56:24.827 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table data_source to data_source_v1 - v1" 2024-04-12T21:56:24.827 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table v2" 2024-04-12T21:56:24.827 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_data_source_org_id - v2" 2024-04-12T21:56:24.827 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_data_source_org_id_name - v2" 2024-04-12T21:56:24.827 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-04-12T21:56:24.827 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table data_source_v1 #2" 2024-04-12T21:56:24.828 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add column with_credentials" 2024-04-12T21:56:24.828 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add secure json data column" 2024-04-12T21:56:24.828 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Update data_source table charset" 2024-04-12T21:56:24.828 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Update initial version to 1" 2024-04-12T21:56:24.828 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add read_only data column" 2024-04-12T21:56:24.828 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate logging ds to loki ds" 2024-04-12T21:56:24.828 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Update json_data with nulls" 2024-04-12T21:56:24.828 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add uid column" 2024-04-12T21:56:24.828 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid value" 2024-04-12T21:56:24.828 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index datasource_org_id_uid" 2024-04-12T21:56:24.828 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index datasource_org_id_is_default" 2024-04-12T21:56:24.828 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table" 2024-04-12T21:56:24.828 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id" 2024-04-12T21:56:24.828 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.key" 2024-04-12T21:56:24.829 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id_name" 2024-04-12T21:56:24.829 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_api_key_account_id - v1" 2024-04-12T21:56:24.829 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_key - v1" 2024-04-12T21:56:24.829 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_account_id_name - v1" 2024-04-12T21:56:24.829 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table api_key to api_key_v1 - v1" 2024-04-12T21:56:24.829 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table v2" 2024-04-12T21:56:24.829 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_api_key_org_id - v2" 2024-04-12T21:56:24.829 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_key - v2" 2024-04-12T21:56:24.829 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_org_id_name - v2" 2024-04-12T21:56:24.829 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="copy api_key v1 to v2" 2024-04-12T21:56:24.829 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table api_key_v1" 2024-04-12T21:56:24.829 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Update api_key table charset" 2024-04-12T21:56:24.829 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add expires to api_key table" 2024-04-12T21:56:24.829 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add service account foreign key" 2024-04-12T21:56:24.830 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v4" 2024-04-12T21:56:24.830 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_snapshot_v4 #1" 2024-04-12T21:56:24.830 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v5 #2" 2024-04-12T21:56:24.830 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_key - v5" 2024-04-12T21:56:24.830 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-04-12T21:56:24.830 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_snapshot_user_id - v5" 2024-04-12T21:56:24.830 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_snapshot to mediumtext v2" 2024-04-12T21:56:24.830 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_snapshot table charset" 2024-04-12T21:56:24.830 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external_delete_url to dashboard_snapshots table" 2024-04-12T21:56:24.831 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add encrypted dashboard json column" 2024-04-12T21:56:24.831 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-04-12T21:56:24.831 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create quota table v1" 2024-04-12T21:56:24.831 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_quota_org_id_user_id_target - v1" 2024-04-12T21:56:24.832 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Update quota table charset" 2024-04-12T21:56:24.832 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create plugin_setting table" 2024-04-12T21:56:24.832 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-04-12T21:56:24.832 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_version to plugin_settings" 2024-04-12T21:56:24.832 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Update plugin_setting table charset" 2024-04-12T21:56:24.832 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create session table" 2024-04-12T21:56:24.832 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist table" 2024-04-12T21:56:24.832 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist_item table" 2024-04-12T21:56:24.832 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist table v2" 2024-04-12T21:56:24.832 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist item table v2" 2024-04-12T21:56:24.832 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist table charset" 2024-04-12T21:56:24.832 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist_item table charset" 2024-04-12T21:56:24.832 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v2" 2024-04-12T21:56:24.832 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v3" 2024-04-12T21:56:24.832 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create preferences table v3" 2024-04-12T21:56:24.833 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Update preferences table charset" 2024-04-12T21:56:24.833 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add column team_id in preferences" 2024-04-12T21:56:24.833 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Update team_id column values in preferences" 2024-04-12T21:56:24.833 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add column week_start in preferences" 2024-04-12T21:56:24.833 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create alert table v1" 2024-04-12T21:56:24.833 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert org_id & id " 2024-04-12T21:56:24.833 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert state" 2024-04-12T21:56:24.833 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert dashboard_id" 2024-04-12T21:56:24.833 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v1" 2024-04-12T21:56:24.833 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-04-12T21:56:24.833 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-04-12T21:56:24.833 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-04-12T21:56:24.833 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v2" 2024-04-12T21:56:24.833 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-04-12T21:56:24.834 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="copy alert_rule_tag v1 to v2" 2024-04-12T21:56:24.834 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop table alert_rule_tag_v1" 2024-04-12T21:56:24.834 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification table v1" 2024-04-12T21:56:24.834 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add column is_default" 2024-04-12T21:56:24.834 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add column frequency" 2024-04-12T21:56:24.834 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add column send_reminder" 2024-04-12T21:56:24.834 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add column disable_resolve_message" 2024-04-12T21:56:24.834 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification org_id & name" 2024-04-12T21:56:24.834 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert table charset" 2024-04-12T21:56:24.834 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert_notification table charset" 2024-04-12T21:56:24.835 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create notification_journal table v1" 2024-04-12T21:56:24.835 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index notification_journal org_id & alert_id & notifier_id" 2024-04-12T21:56:24.835 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_notification_journal" 2024-04-12T21:56:24.835 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification_state table v1" 2024-04-12T21:56:24.835 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-04-12T21:56:24.835 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add for to alert table" 2024-04-12T21:56:24.835 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in alert_notification" 2024-04-12T21:56:24.835 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in alert_notification" 2024-04-12T21:56:24.835 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_notification_org_id_uid" 2024-04-12T21:56:24.835 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_name" 2024-04-12T21:56:24.835 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add column secure_settings in alert_notification" 2024-04-12T21:56:24.835 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert.settings to mediumtext" 2024-04-12T21:56:24.836 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_notification_state_alert_id" 2024-04-12T21:56:24.836 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_rule_tag_alert_id" 2024-04-12T21:56:24.836 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old annotation table v4" 2024-04-12T21:56:24.836 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create annotation table v5" 2024-04-12T21:56:24.836 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 0 v3" 2024-04-12T21:56:24.836 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 1 v3" 2024-04-12T21:56:24.836 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 2 v3" 2024-04-12T21:56:24.836 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 3 v3" 2024-04-12T21:56:24.836 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 4 v3" 2024-04-12T21:56:24.836 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Update annotation table charset" 2024-04-12T21:56:24.836 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add column region_id to annotation table" 2024-04-12T21:56:24.837 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Drop category_id index" 2024-04-12T21:56:24.837 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add column tags to annotation table" 2024-04-12T21:56:24.837 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v2" 2024-04-12T21:56:24.837 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index annotation_tag.annotation_id_tag_id" 2024-04-12T21:56:24.837 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-04-12T21:56:24.837 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-04-12T21:56:24.837 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v3" 2024-04-12T21:56:24.837 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-04-12T21:56:24.837 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="copy annotation_tag v2 to v3" 2024-04-12T21:56:24.837 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop table annotation_tag_v2" 2024-04-12T21:56:24.837 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert annotations and set TEXT to empty" 2024-04-12T21:56:24.838 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add created time to annotation table" 2024-04-12T21:56:24.838 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add updated time to annotation table" 2024-04-12T21:56:24.838 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for created in annotation table" 2024-04-12T21:56:24.838 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for updated in annotation table" 2024-04-12T21:56:24.838 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Convert existing annotations from seconds to milliseconds" 2024-04-12T21:56:24.838 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add epoch_end column" 2024-04-12T21:56:24.838 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for epoch_end" 2024-04-12T21:56:24.838 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Make epoch_end the same as epoch" 2024-04-12T21:56:24.838 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Move region to single row" 2024-04-12T21:56:24.838 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch from annotation table" 2024-04-12T21:56:24.838 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-04-12T21:56:24.838 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-04-12T21:56:24.838 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_epoch_end_epoch on annotation table" 2024-04-12T21:56:24.838 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch_epoch_end from annotation table" 2024-04-12T21:56:24.839 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for alert_id on annotation table" 2024-04-12T21:56:24.839 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create test_data table" 2024-04-12T21:56:24.839 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_version table v1" 2024-04-12T21:56:24.839 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_version.dashboard_id" 2024-04-12T21:56:24.839 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-04-12T21:56:24.839 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Set dashboard version to 1 where 0" 2024-04-12T21:56:24.839 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="save existing dashboard data in dashboard_version table v1" 2024-04-12T21:56:24.839 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_version.data to mediumtext v1" 2024-04-12T21:56:24.839 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create team table" 2024-04-12T21:56:24.839 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index team.org_id" 2024-04-12T21:56:24.839 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_org_id_name" 2024-04-12T21:56:24.839 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create team member table" 2024-04-12T21:56:24.839 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.org_id" 2024-04-12T21:56:25.040 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:24 smithi097 ceph-mon[24835]: osdmap e60: 8 total, 8 up, 8 in 2024-04-12T21:56:25.040 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:24 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/2184930125' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-12T21:56:25.040 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:24 smithi097 ceph-mon[24835]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-12T21:56:25.041 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:24 smithi097 ceph-mon[29620]: osdmap e60: 8 total, 8 up, 8 in 2024-04-12T21:56:25.041 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:24 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/2184930125' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-12T21:56:25.041 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:24 smithi097 ceph-mon[29620]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-12T21:56:25.072 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_member_org_id_team_id_user_id" 2024-04-12T21:56:25.072 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.team_id" 2024-04-12T21:56:25.072 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add column email to team table" 2024-04-12T21:56:25.072 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external to team_member table" 2024-04-12T21:56:25.072 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add column permission to team_member table" 2024-04-12T21:56:25.073 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard acl table" 2024-04-12T21:56:25.073 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_dashboard_id" 2024-04-12T21:56:25.073 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_user_id" 2024-04-12T21:56:25.073 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_team_id" 2024-04-12T21:56:25.073 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_user_id" 2024-04-12T21:56:25.073 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_team_id" 2024-04-12T21:56:25.073 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_org_id_role" 2024-04-12T21:56:25.073 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_permission" 2024-04-12T21:56:25.073 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="save default acl rules in dashboard_acl table" 2024-04-12T21:56:25.073 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="delete acl rules for deleted dashboards and folders" 2024-04-12T21:56:25.073 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create tag table" 2024-04-12T21:56:25.074 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index tag.key_value" 2024-04-12T21:56:25.074 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create login attempt table" 2024-04-12T21:56:25.074 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index login_attempt.username" 2024-04-12T21:56:25.075 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_login_attempt_username - v1" 2024-04-12T21:56:25.075 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-04-12T21:56:25.075 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create login_attempt v2" 2024-04-12T21:56:25.075 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_login_attempt_username - v2" 2024-04-12T21:56:25.075 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="copy login_attempt v1 to v2" 2024-04-12T21:56:25.075 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop login_attempt_tmp_qwerty" 2024-04-12T21:56:25.075 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth table" 2024-04-12T21:56:25.075 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-04-12T21:56:25.076 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="alter user_auth.auth_id to length 190" 2024-04-12T21:56:25.076 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth access token to user_auth" 2024-04-12T21:56:25.076 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth refresh token to user_auth" 2024-04-12T21:56:25.076 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth token type to user_auth" 2024-04-12T21:56:25.076 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth expiry to user_auth" 2024-04-12T21:56:25.076 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add index to user_id column in user_auth" 2024-04-12T21:56:25.076 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create server_lock table" 2024-04-12T21:56:25.076 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index server_lock.operation_uid" 2024-04-12T21:56:25.076 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth token table" 2024-04-12T21:56:25.076 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.auth_token" 2024-04-12T21:56:25.076 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.prev_auth_token" 2024-04-12T21:56:25.077 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_auth_token.user_id" 2024-04-12T21:56:25.077 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add revoked_at to the user auth token" 2024-04-12T21:56:25.077 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create cache_data table" 2024-04-12T21:56:25.077 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index cache_data.cache_key" 2024-04-12T21:56:25.077 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create short_url table v1" 2024-04-12T21:56:25.077 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index short_url.org_id-uid" 2024-04-12T21:56:25.077 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition table" 2024-04-12T21:56:25.077 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition table" 2024-04-12T21:56:25.077 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and title columns" 2024-04-12T21:56:25.077 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and uid columns" 2024-04-12T21:56:25.077 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition table data column to mediumtext in mysql" 2024-04-12T21:56:25.077 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and title columns" 2024-04-12T21:56:25.078 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and uid columns" 2024-04-12T21:56:25.078 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and title columns" 2024-04-12T21:56:25.078 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and uid columns" 2024-04-12T21:56:25.078 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add column paused in alert_definition" 2024-04-12T21:56:25.078 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition table" 2024-04-12T21:56:25.078 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition_version table" 2024-04-12T21:56:25.078 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition_version table" 2024-04-12T21:56:25.078 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-04-12T21:56:25.078 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-04-12T21:56:25.078 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition_version table data column to mediumtext in mysql" 2024-04-12T21:56:25.078 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition_version table" 2024-04-12T21:56:25.079 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_instance table" 2024-04-12T21:56:25.079 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-04-12T21:56:25.079 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, current_state columns" 2024-04-12T21:56:25.079 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add column current_state_end to alert_instance" 2024-04-12T21:56:25.079 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-04-12T21:56:25.079 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, current_state on alert_instance" 2024-04-12T21:56:25.079 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_org_id to rule_org_id in alert_instance" 2024-04-12T21:56:25.079 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_uid to rule_uid in alert_instance" 2024-04-12T21:56:25.079 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-04-12T21:56:25.079 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, current_state on alert_instance" 2024-04-12T21:56:25.079 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule table" 2024-04-12T21:56:25.079 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and title columns" 2024-04-12T21:56:25.080 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and uid columns" 2024-04-12T21:56:25.080 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-04-12T21:56:25.080 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule table data column to mediumtext in mysql" 2024-04-12T21:56:25.080 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule" 2024-04-12T21:56:25.080 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule" 2024-04-12T21:56:25.080 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule" 2024-04-12T21:56:25.080 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="remove unique index from alert_rule on org_id, title columns" 2024-04-12T21:56:25.080 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-04-12T21:56:25.080 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add dashboard_uid column to alert_rule" 2024-04-12T21:56:25.080 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add panel_id column to alert_rule" 2024-04-12T21:56:25.080 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-04-12T21:56:25.080 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule_version table" 2024-04-12T21:56:25.081 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-04-12T21:56:25.081 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-04-12T21:56:25.081 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule_version table data column to mediumtext in mysql" 2024-04-12T21:56:25.081 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule_version" 2024-04-12T21:56:25.081 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule_version" 2024-04-12T21:56:25.081 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule_version" 2024-04-12T21:56:25.081 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id=create_alert_configuration_table 2024-04-12T21:56:25.081 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="Add column default in alert_configuration" 2024-04-12T21:56:25.081 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-04-12T21:56:25.081 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add column org_id in alert_configuration" 2024-04-12T21:56:25.082 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_configuration table on org_id column" 2024-04-12T21:56:25.082 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id=create_ngalert_configuration_table 2024-04-12T21:56:25.082 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index in ngalert_configuration on org_id column" 2024-04-12T21:56:25.082 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="clear migration entry \"remove unified alerting data\"" 2024-04-12T21:56:25.082 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="move dashboard alerts to unified alerting" 2024-04-12T21:56:25.083 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element table v1" 2024-04-12T21:56:25.083 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element org_id-folder_id-name-kind" 2024-04-12T21:56:25.083 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element_connection table v1" 2024-04-12T21:56:25.083 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element_connection element_id-kind-connection_id" 2024-04-12T21:56:25.083 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index library_element org_id_uid" 2024-04-12T21:56:25.083 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="clone move dashboard alerts to unified alerting" 2024-04-12T21:56:25.083 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create data_keys table" 2024-04-12T21:56:25.083 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create kv_store table v1" 2024-04-12T21:56:25.083 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index kv_store.org_id-namespace-key" 2024-04-12T21:56:25.083 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="update dashboard_uid and panel_id from existing annotations" 2024-04-12T21:56:25.083 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create permission table" 2024-04-12T21:56:25.084 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index permission.role_id" 2024-04-12T21:56:25.084 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_id_action_scope" 2024-04-12T21:56:25.084 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create role table" 2024-04-12T21:56:25.084 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add column display_name" 2024-04-12T21:56:25.084 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add column group_name" 2024-04-12T21:56:25.084 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index role.org_id" 2024-04-12T21:56:25.084 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_org_id_name" 2024-04-12T21:56:25.084 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="add index role_org_id_uid" 2024-04-12T21:56:25.084 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:24+0000 lvl=info msg="Executing migration" logger=migrator id="create team role table" 2024-04-12T21:56:25.084 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:25 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:25+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.org_id" 2024-04-12T21:56:25.084 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:25 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:25+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_role_org_id_team_id_role_id" 2024-04-12T21:56:25.084 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:25 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:25+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.team_id" 2024-04-12T21:56:25.085 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:25 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:25+0000 lvl=info msg="Executing migration" logger=migrator id="create user role table" 2024-04-12T21:56:25.085 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:25 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:25+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.org_id" 2024-04-12T21:56:25.085 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:25 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:25+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_role_org_id_user_id_role_id" 2024-04-12T21:56:25.085 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:25 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:25+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.user_id" 2024-04-12T21:56:25.085 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:25 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:25+0000 lvl=info msg="Executing migration" logger=migrator id="create builtin role table" 2024-04-12T21:56:25.085 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:25 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:25+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.role_id" 2024-04-12T21:56:25.085 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:25 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:25+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.name" 2024-04-12T21:56:25.085 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:25 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:25+0000 lvl=info msg="Executing migration" logger=migrator id="Add column org_id to builtin_role table" 2024-04-12T21:56:25.085 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:25 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:25+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.org_id" 2024-04-12T21:56:25.085 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:25 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:25+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_org_id_role_id_role" 2024-04-12T21:56:25.085 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:25 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:25+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index role_org_id_uid" 2024-04-12T21:56:25.085 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:25 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:25+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role.uid" 2024-04-12T21:56:25.086 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:25 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:25+0000 lvl=info msg="Executing migration" logger=migrator id="create seed assignment table" 2024-04-12T21:56:25.086 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:25 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:25+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_role_name" 2024-04-12T21:56:25.086 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:25 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:25+0000 lvl=info msg="migrations completed" logger=migrator performed=381 skipped=0 duration=451.931581ms 2024-04-12T21:56:25.086 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:25 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:25+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-04-12T21:56:25.086 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:25 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:25+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2024-04-12T21:56:25.086 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:25 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:25+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2024-04-12T21:56:25.086 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:25 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:25+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2024-04-12T21:56:25.086 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:25 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:25+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2024-04-12T21:56:25.086 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:25 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:25+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2024-04-12T21:56:25.363 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -e sha1=b277feb2c7beb4597d8ecc39d58d5c36b81f613a -- bash -c 'radosgw-admin zonegroup create --rgw-zonegroup=default --master --default' 2024-04-12T21:56:25.401 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:25 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:25+0000 lvl=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" logger=provisioning.datasources filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2024-04-12T21:56:25.402 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:25 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:25+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2024-04-12T21:56:25.402 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:25 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:25+0000 lvl=info msg="warming cache for startup" logger=ngalert 2024-04-12T21:56:25.402 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:25 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:25+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-04-12T21:56:25.402 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 21:56:25 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T21:56:25+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2024-04-12T21:56:25.673 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:25 smithi114 ceph-mon[25759]: pgmap v46: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:56:25.673 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:25 smithi114 ceph-mon[25759]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-04-12T21:56:25.673 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:25 smithi114 ceph-mon[25759]: osdmap e61: 8 total, 8 up, 8 in 2024-04-12T21:56:25.673 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:25 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:25.952 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:25 smithi097 ceph-mon[24835]: pgmap v46: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:56:25.952 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:25 smithi097 ceph-mon[24835]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-04-12T21:56:25.952 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:25 smithi097 ceph-mon[24835]: osdmap e61: 8 total, 8 up, 8 in 2024-04-12T21:56:25.952 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:25 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:25.952 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:25 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:56:25.952 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:25 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:56:25.952 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:25 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:25.953 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:25 smithi097 ceph-mon[29620]: pgmap v46: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:56:25.953 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:25 smithi097 ceph-mon[29620]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-04-12T21:56:25.953 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:25 smithi097 ceph-mon[29620]: osdmap e61: 8 total, 8 up, 8 in 2024-04-12T21:56:25.953 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:25 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:25.953 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:25 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:56:25.953 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:25 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:56:25.953 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:25 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:25.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:25 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:56:25.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:25 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:56:25.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:25 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:26.319 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:56:26 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: ::ffff:172.21.15.114 - - [12/Apr/2024:21:56:26] "GET /metrics HTTP/1.1" 200 191499 "" "Prometheus/2.33.4" 2024-04-12T21:56:26.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:26 smithi097 ceph-mon[24835]: osdmap e62: 8 total, 8 up, 8 in 2024-04-12T21:56:26.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:26 smithi097 ceph-mon[29620]: osdmap e62: 8 total, 8 up, 8 in 2024-04-12T21:56:26.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:26 smithi114 ceph-mon[25759]: osdmap e62: 8 total, 8 up, 8 in 2024-04-12T21:56:27.713 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-12T21:56:27.714 INFO:teuthology.orchestra.run.smithi097.stdout: "id": "3879b12e-8b81-4a17-9f55-c05bead7a73d", 2024-04-12T21:56:27.714 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "default", 2024-04-12T21:56:27.714 INFO:teuthology.orchestra.run.smithi097.stdout: "api_name": "default", 2024-04-12T21:56:27.714 INFO:teuthology.orchestra.run.smithi097.stdout: "is_master": "true", 2024-04-12T21:56:27.714 INFO:teuthology.orchestra.run.smithi097.stdout: "endpoints": [], 2024-04-12T21:56:27.714 INFO:teuthology.orchestra.run.smithi097.stdout: "hostnames": [], 2024-04-12T21:56:27.714 INFO:teuthology.orchestra.run.smithi097.stdout: "hostnames_s3website": [], 2024-04-12T21:56:27.714 INFO:teuthology.orchestra.run.smithi097.stdout: "master_zone": "", 2024-04-12T21:56:27.714 INFO:teuthology.orchestra.run.smithi097.stdout: "zones": [], 2024-04-12T21:56:27.714 INFO:teuthology.orchestra.run.smithi097.stdout: "placement_targets": [], 2024-04-12T21:56:27.714 INFO:teuthology.orchestra.run.smithi097.stdout: "default_placement": "", 2024-04-12T21:56:27.714 INFO:teuthology.orchestra.run.smithi097.stdout: "realm_id": "8e06d417-cc67-4293-b394-c4ab3e41b979", 2024-04-12T21:56:27.715 INFO:teuthology.orchestra.run.smithi097.stdout: "sync_policy": { 2024-04-12T21:56:27.715 INFO:teuthology.orchestra.run.smithi097.stdout: "groups": [] 2024-04-12T21:56:27.715 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-12T21:56:27.715 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-12T21:56:27.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:27 smithi097 ceph-mon[24835]: pgmap v49: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:56:27.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:27 smithi097 ceph-mon[29620]: pgmap v49: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:56:27.961 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:27 smithi114 ceph-mon[25759]: pgmap v49: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:56:28.424 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -e sha1=b277feb2c7beb4597d8ecc39d58d5c36b81f613a -- bash -c 'radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default' 2024-04-12T21:56:29.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:28 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:29.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:28 smithi097 ceph-mon[29620]: pgmap v50: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-04-12T21:56:29.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:28 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:29.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:28 smithi097 ceph-mon[24835]: pgmap v50: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-04-12T21:56:29.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:28 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:29.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:28 smithi114 ceph-mon[25759]: pgmap v50: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-04-12T21:56:31.478 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-12T21:56:31.478 INFO:teuthology.orchestra.run.smithi097.stdout: "id": "d22b1b60-1034-4324-aec2-c7a1b5510ffa", 2024-04-12T21:56:31.478 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "z", 2024-04-12T21:56:31.478 INFO:teuthology.orchestra.run.smithi097.stdout: "domain_root": "z.rgw.meta:root", 2024-04-12T21:56:31.478 INFO:teuthology.orchestra.run.smithi097.stdout: "control_pool": "z.rgw.control", 2024-04-12T21:56:31.478 INFO:teuthology.orchestra.run.smithi097.stdout: "gc_pool": "z.rgw.log:gc", 2024-04-12T21:56:31.478 INFO:teuthology.orchestra.run.smithi097.stdout: "lc_pool": "z.rgw.log:lc", 2024-04-12T21:56:31.478 INFO:teuthology.orchestra.run.smithi097.stdout: "log_pool": "z.rgw.log", 2024-04-12T21:56:31.478 INFO:teuthology.orchestra.run.smithi097.stdout: "intent_log_pool": "z.rgw.log:intent", 2024-04-12T21:56:31.478 INFO:teuthology.orchestra.run.smithi097.stdout: "usage_log_pool": "z.rgw.log:usage", 2024-04-12T21:56:31.478 INFO:teuthology.orchestra.run.smithi097.stdout: "roles_pool": "z.rgw.meta:roles", 2024-04-12T21:56:31.478 INFO:teuthology.orchestra.run.smithi097.stdout: "reshard_pool": "z.rgw.log:reshard", 2024-04-12T21:56:31.478 INFO:teuthology.orchestra.run.smithi097.stdout: "user_keys_pool": "z.rgw.meta:users.keys", 2024-04-12T21:56:31.479 INFO:teuthology.orchestra.run.smithi097.stdout: "user_email_pool": "z.rgw.meta:users.email", 2024-04-12T21:56:31.479 INFO:teuthology.orchestra.run.smithi097.stdout: "user_swift_pool": "z.rgw.meta:users.swift", 2024-04-12T21:56:31.479 INFO:teuthology.orchestra.run.smithi097.stdout: "user_uid_pool": "z.rgw.meta:users.uid", 2024-04-12T21:56:31.479 INFO:teuthology.orchestra.run.smithi097.stdout: "otp_pool": "z.rgw.otp", 2024-04-12T21:56:31.479 INFO:teuthology.orchestra.run.smithi097.stdout: "system_key": { 2024-04-12T21:56:31.479 INFO:teuthology.orchestra.run.smithi097.stdout: "access_key": "", 2024-04-12T21:56:31.479 INFO:teuthology.orchestra.run.smithi097.stdout: "secret_key": "" 2024-04-12T21:56:31.479 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T21:56:31.479 INFO:teuthology.orchestra.run.smithi097.stdout: "placement_pools": [ 2024-04-12T21:56:31.479 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-12T21:56:31.479 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "default-placement", 2024-04-12T21:56:31.479 INFO:teuthology.orchestra.run.smithi097.stdout: "val": { 2024-04-12T21:56:31.479 INFO:teuthology.orchestra.run.smithi097.stdout: "index_pool": "z.rgw.buckets.index", 2024-04-12T21:56:31.479 INFO:teuthology.orchestra.run.smithi097.stdout: "storage_classes": { 2024-04-12T21:56:31.479 INFO:teuthology.orchestra.run.smithi097.stdout: "STANDARD": { 2024-04-12T21:56:31.479 INFO:teuthology.orchestra.run.smithi097.stdout: "data_pool": "z.rgw.buckets.data" 2024-04-12T21:56:31.480 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-12T21:56:31.480 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T21:56:31.480 INFO:teuthology.orchestra.run.smithi097.stdout: "data_extra_pool": "z.rgw.buckets.non-ec", 2024-04-12T21:56:31.480 INFO:teuthology.orchestra.run.smithi097.stdout: "index_type": 0 2024-04-12T21:56:31.480 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-12T21:56:31.480 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-12T21:56:31.480 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-04-12T21:56:31.480 INFO:teuthology.orchestra.run.smithi097.stdout: "realm_id": "8e06d417-cc67-4293-b394-c4ab3e41b979", 2024-04-12T21:56:31.480 INFO:teuthology.orchestra.run.smithi097.stdout: "notif_pool": "z.rgw.log:notif" 2024-04-12T21:56:31.480 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-12T21:56:31.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:31 smithi097 ceph-mon[24835]: pgmap v51: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 1 op/s 2024-04-12T21:56:31.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:31 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:31.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:31 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:31.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:31 smithi097 ceph-mon[29620]: pgmap v51: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 1 op/s 2024-04-12T21:56:31.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:31 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:31.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:31 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:31.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:31 smithi114 ceph-mon[25759]: pgmap v51: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 1 op/s 2024-04-12T21:56:31.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:31 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:31.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:31 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:32.055 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -e sha1=b277feb2c7beb4597d8ecc39d58d5c36b81f613a -- bash -c 'radosgw-admin period update --rgw-realm=r --commit' 2024-04-12T21:56:32.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:32 smithi097 ceph-mon[24835]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-12T21:56:32.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:32 smithi097 ceph-mon[24835]: Reconfiguring daemon alertmanager.a on smithi097 2024-04-12T21:56:32.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:32 smithi097 ceph-mon[29620]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-12T21:56:32.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:32 smithi097 ceph-mon[29620]: Reconfiguring daemon alertmanager.a on smithi097 2024-04-12T21:56:32.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:32 smithi114 ceph-mon[25759]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-12T21:56:32.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:32 smithi114 ceph-mon[25759]: Reconfiguring daemon alertmanager.a on smithi097 2024-04-12T21:56:33.776 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:33 smithi097 ceph-mon[24835]: pgmap v52: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 2.0 KiB/s wr, 3 op/s 2024-04-12T21:56:33.776 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:33 smithi097 ceph-mon[29620]: pgmap v52: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 2.0 KiB/s wr, 3 op/s 2024-04-12T21:56:33.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:33 smithi114 ceph-mon[25759]: pgmap v52: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 2.0 KiB/s wr, 3 op/s 2024-04-12T21:56:34.744 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:56:34 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: ::ffff:172.21.15.114 - - [12/Apr/2024:21:56:34] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-12T21:56:35.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:35 smithi097 ceph-mon[24835]: pgmap v53: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 721 B/s rd, 1.6 KiB/s wr, 2 op/s 2024-04-12T21:56:35.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:35 smithi097 ceph-mon[29620]: pgmap v53: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 721 B/s rd, 1.6 KiB/s wr, 2 op/s 2024-04-12T21:56:35.912 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 21:56:35 smithi097 systemd[1]: Stopping Ceph alertmanager.a for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T21:56:35.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:35 smithi114 ceph-mon[25759]: pgmap v53: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 721 B/s rd, 1.6 KiB/s wr, 2 op/s 2024-04-12T21:56:36.378 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:56:36 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: ::ffff:172.21.15.114 - - [12/Apr/2024:21:56:36] "GET /metrics HTTP/1.1" 200 192867 "" "Prometheus/2.33.4" 2024-04-12T21:56:36.653 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 21:56:36 smithi097 bash[49470]: Error: no container with name or ID "ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager.a" found: no such container 2024-04-12T21:56:36.911 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 21:56:36 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[47012]: level=info ts=2024-04-12T21:56:36.855Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-04-12T21:56:36.911 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 21:56:36 smithi097 podman[49507]: 2024-04-12 21:56:36.876215792 +0000 UTC m=+0.479342496 container died 21406e986e92db067752f3e8ca5737c27900fa8672bc6b3b6c4429fbeb59e865 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-12T21:56:37.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:36 smithi114 ceph-mon[25759]: pgmap v54: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 656 B/s rd, 1.5 KiB/s wr, 2 op/s 2024-04-12T21:56:37.257 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:36 smithi097 ceph-mon[24835]: pgmap v54: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 656 B/s rd, 1.5 KiB/s wr, 2 op/s 2024-04-12T21:56:37.257 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:36 smithi097 ceph-mon[29620]: pgmap v54: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 656 B/s rd, 1.5 KiB/s wr, 2 op/s 2024-04-12T21:56:37.525 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 21:56:37 smithi097 podman[49507]: 2024-04-12 21:56:37.256141513 +0000 UTC m=+0.859268213 container cleanup 21406e986e92db067752f3e8ca5737c27900fa8672bc6b3b6c4429fbeb59e865 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-12T21:56:37.525 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 21:56:37 smithi097 bash[49507]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a 2024-04-12T21:56:37.525 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 21:56:37 smithi097 bash[49590]: Error: no container with name or ID "ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager.a" found: no such container 2024-04-12T21:56:37.860 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 21:56:37 smithi097 podman[49574]: 2024-04-12 21:56:37.524126796 +0000 UTC m=+0.647599322 container remove 21406e986e92db067752f3e8ca5737c27900fa8672bc6b3b6c4429fbeb59e865 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-12T21:56:37.860 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 21:56:37 smithi097 podman[49574]: 2024-04-12 21:56:37.641495969 +0000 UTC m=+0.764968494 volume remove 5f0abaf5f4a3a4277c08023cb93afccf8acaf85da8a66a4160a67dd8d703c199 2024-04-12T21:56:37.860 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 21:56:37 smithi097 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-04-12T21:56:37.860 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 21:56:37 smithi097 systemd[1]: Stopped Ceph alertmanager.a for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T21:56:37.860 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 21:56:37 smithi097 systemd[1]: Starting Ceph alertmanager.a for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T21:56:38.156 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 21:56:37 smithi097 podman[49661]: 2024-04-12 21:56:37.859095914 +0000 UTC m=+0.109980650 volume create 6c7e166be94e9c6a41dedbab10931a486570c7d56c3abd315737938621ab0be5 2024-04-12T21:56:38.156 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 21:56:37 smithi097 podman[49661]: 2024-04-12 21:56:37.766888422 +0000 UTC m=+0.017773158 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-04-12T21:56:38.156 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 21:56:37 smithi097 podman[49661]: 2024-04-12 21:56:37.951212699 +0000 UTC m=+0.202097482 container create 756b0c8c01e197e3da2e10cd7e1e6b572bc1c02bb51c96a15421505db564bc25 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-12T21:56:38.156 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:37 smithi097 ceph-mon[24835]: osdmap e63: 8 total, 8 up, 8 in 2024-04-12T21:56:38.156 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:37 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/2329841363' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-04-12T21:56:38.156 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:37 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/2329841363' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-04-12T21:56:38.156 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:37 smithi097 ceph-mon[24835]: osdmap e64: 8 total, 8 up, 8 in 2024-04-12T21:56:38.157 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:37 smithi097 ceph-mon[29620]: osdmap e63: 8 total, 8 up, 8 in 2024-04-12T21:56:38.157 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:37 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/2329841363' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-04-12T21:56:38.157 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:37 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/2329841363' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-04-12T21:56:38.157 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:37 smithi097 ceph-mon[29620]: osdmap e64: 8 total, 8 up, 8 in 2024-04-12T21:56:38.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:37 smithi114 ceph-mon[25759]: osdmap e63: 8 total, 8 up, 8 in 2024-04-12T21:56:38.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:37 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/2329841363' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-04-12T21:56:38.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:37 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/2329841363' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-04-12T21:56:38.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:37 smithi114 ceph-mon[25759]: osdmap e64: 8 total, 8 up, 8 in 2024-04-12T21:56:38.661 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 21:56:38 smithi097 podman[49661]: 2024-04-12 21:56:38.287811611 +0000 UTC m=+0.538696347 container init 756b0c8c01e197e3da2e10cd7e1e6b572bc1c02bb51c96a15421505db564bc25 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-12T21:56:38.661 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 21:56:38 smithi097 podman[49661]: 2024-04-12 21:56:38.291093666 +0000 UTC m=+0.541978405 container start 756b0c8c01e197e3da2e10cd7e1e6b572bc1c02bb51c96a15421505db564bc25 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-12T21:56:38.662 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 21:56:38 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[49677]: level=info ts=2024-04-12T21:56:38.300Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-04-12T21:56:38.662 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 21:56:38 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[49677]: level=info ts=2024-04-12T21:56:38.301Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-04-12T21:56:38.662 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 21:56:38 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[49677]: level=info ts=2024-04-12T21:56:38.302Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.97 port=9094 2024-04-12T21:56:38.662 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 21:56:38 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[49677]: level=info ts=2024-04-12T21:56:38.303Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-04-12T21:56:38.662 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 21:56:38 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[49677]: level=info ts=2024-04-12T21:56:38.327Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-12T21:56:38.662 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 21:56:38 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[49677]: level=info ts=2024-04-12T21:56:38.327Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-12T21:56:38.662 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 21:56:38 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[49677]: level=info ts=2024-04-12T21:56:38.330Z caller=main.go:518 msg=Listening address=:9093 2024-04-12T21:56:38.662 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 21:56:38 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[49677]: level=info ts=2024-04-12T21:56:38.330Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-04-12T21:56:38.662 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 21:56:38 smithi097 bash[49661]: 756b0c8c01e197e3da2e10cd7e1e6b572bc1c02bb51c96a15421505db564bc25 2024-04-12T21:56:38.662 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 21:56:38 smithi097 systemd[1]: Started Ceph alertmanager.a for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T21:56:39.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:39 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:39.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:39 smithi114 ceph-mon[25759]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-12T21:56:39.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:39 smithi114 ceph-mon[25759]: Reconfiguring daemon prometheus.a on smithi114 2024-04-12T21:56:39.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:39 smithi114 ceph-mon[25759]: pgmap v57: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 1023 B/s wr, 2 op/s 2024-04-12T21:56:39.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:39 smithi114 ceph-mon[25759]: osdmap e65: 8 total, 8 up, 8 in 2024-04-12T21:56:39.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:39 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/2329841363' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-04-12T21:56:39.799 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:39 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:39.799 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:39 smithi097 ceph-mon[29620]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-12T21:56:39.799 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:39 smithi097 ceph-mon[29620]: Reconfiguring daemon prometheus.a on smithi114 2024-04-12T21:56:39.799 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:39 smithi097 ceph-mon[29620]: pgmap v57: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 1023 B/s wr, 2 op/s 2024-04-12T21:56:39.799 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:39 smithi097 ceph-mon[29620]: osdmap e65: 8 total, 8 up, 8 in 2024-04-12T21:56:39.799 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:39 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/2329841363' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-04-12T21:56:39.799 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:39 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:39.799 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:39 smithi097 ceph-mon[24835]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-12T21:56:39.799 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:39 smithi097 ceph-mon[24835]: Reconfiguring daemon prometheus.a on smithi114 2024-04-12T21:56:39.799 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:39 smithi097 ceph-mon[24835]: pgmap v57: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 1023 B/s wr, 2 op/s 2024-04-12T21:56:39.800 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:39 smithi097 ceph-mon[24835]: osdmap e65: 8 total, 8 up, 8 in 2024-04-12T21:56:39.800 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:39 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/2329841363' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-04-12T21:56:40.661 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 21:56:40 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[49677]: level=info ts=2024-04-12T21:56:40.303Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000414874s 2024-04-12T21:56:41.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:40 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/2329841363' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-04-12T21:56:41.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:40 smithi097 ceph-mon[24835]: osdmap e66: 8 total, 8 up, 8 in 2024-04-12T21:56:41.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:40 smithi097 ceph-mon[24835]: pgmap v60: 97 pgs: 64 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:56:41.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:40 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/2329841363' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-04-12T21:56:41.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:40 smithi097 ceph-mon[29620]: osdmap e66: 8 total, 8 up, 8 in 2024-04-12T21:56:41.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:40 smithi097 ceph-mon[29620]: pgmap v60: 97 pgs: 64 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:56:41.227 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:40 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/2329841363' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-04-12T21:56:41.227 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:40 smithi114 ceph-mon[25759]: osdmap e66: 8 total, 8 up, 8 in 2024-04-12T21:56:41.227 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:40 smithi114 ceph-mon[25759]: pgmap v60: 97 pgs: 64 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:56:41.781 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:56:41 smithi114 systemd[1]: Stopping Ceph prometheus.a for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T21:56:41.781 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:56:41 smithi114 bash[42527]: Error: no container with name or ID "ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus.a" found: no such container 2024-04-12T21:56:42.105 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:41 smithi114 ceph-mon[25759]: osdmap e67: 8 total, 8 up, 8 in 2024-04-12T21:56:42.106 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:56:41 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[40860]: ts=2024-04-12T21:56:41.779Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-04-12T21:56:42.106 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:56:41 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[40860]: ts=2024-04-12T21:56:41.780Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-04-12T21:56:42.106 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:56:41 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[40860]: ts=2024-04-12T21:56:41.780Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-04-12T21:56:42.106 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:56:41 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[40860]: ts=2024-04-12T21:56:41.780Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-04-12T21:56:42.106 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:56:41 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[40860]: ts=2024-04-12T21:56:41.780Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-04-12T21:56:42.106 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:56:41 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[40860]: ts=2024-04-12T21:56:41.780Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-04-12T21:56:42.106 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:56:41 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[40860]: ts=2024-04-12T21:56:41.780Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-04-12T21:56:42.106 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:56:41 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[40860]: ts=2024-04-12T21:56:41.780Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-04-12T21:56:42.106 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:56:41 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[40860]: ts=2024-04-12T21:56:41.780Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-04-12T21:56:42.106 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:56:41 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[40860]: ts=2024-04-12T21:56:41.781Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-04-12T21:56:42.107 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:56:41 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[40860]: ts=2024-04-12T21:56:41.781Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-04-12T21:56:42.107 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:56:41 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[40860]: ts=2024-04-12T21:56:41.781Z caller=main.go:1066 level=info msg="See you next time!" 2024-04-12T21:56:42.107 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:56:41 smithi114 podman[42540]: 2024-04-12 21:56:41.790117946 +0000 UTC m=+0.107974937 container died df709a0468dddca7d46706b76c467f1ca18faeb24cd18f6b67f70edd1ca7fd35 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-12T21:56:42.107 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:56:42 smithi114 podman[42540]: 2024-04-12 21:56:42.104539586 +0000 UTC m=+0.422396582 container cleanup df709a0468dddca7d46706b76c467f1ca18faeb24cd18f6b67f70edd1ca7fd35 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-12T21:56:42.107 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:56:42 smithi114 bash[42540]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a 2024-04-12T21:56:42.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:41 smithi097 ceph-mon[24835]: osdmap e67: 8 total, 8 up, 8 in 2024-04-12T21:56:42.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:41 smithi097 ceph-mon[29620]: osdmap e67: 8 total, 8 up, 8 in 2024-04-12T21:56:42.494 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:56:42 smithi114 bash[42566]: Error: no container with name or ID "ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus.a" found: no such container 2024-04-12T21:56:42.494 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:56:42 smithi114 podman[42554]: 2024-04-12 21:56:42.322255192 +0000 UTC m=+0.529623019 container remove df709a0468dddca7d46706b76c467f1ca18faeb24cd18f6b67f70edd1ca7fd35 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-12T21:56:42.494 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:56:42 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-04-12T21:56:42.494 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:56:42 smithi114 systemd[1]: Stopped Ceph prometheus.a for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T21:56:42.494 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:56:42 smithi114 systemd[1]: Starting Ceph prometheus.a for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T21:56:42.826 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:56:42 smithi114 podman[42634]: 2024-04-12 21:56:42.456327708 +0000 UTC m=+0.019118561 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-04-12T21:56:42.827 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:56:42 smithi114 podman[42634]: 2024-04-12 21:56:42.606904695 +0000 UTC m=+0.169695535 container create 7a2b6dbe5d8b26479ac240c59da735275ab236572bc449f90775450eb416b4f9 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-12T21:56:43.094 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:42 smithi114 ceph-mon[25759]: osdmap e68: 8 total, 8 up, 8 in 2024-04-12T21:56:43.094 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:42 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/3870310708' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-04-12T21:56:43.094 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:42 smithi114 ceph-mon[25759]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-04-12T21:56:43.094 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:42 smithi114 ceph-mon[25759]: pgmap v63: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-12T21:56:43.094 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:56:42 smithi114 podman[42634]: 2024-04-12 21:56:42.825828587 +0000 UTC m=+0.388619430 container init 7a2b6dbe5d8b26479ac240c59da735275ab236572bc449f90775450eb416b4f9 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-12T21:56:43.094 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:56:42 smithi114 podman[42634]: 2024-04-12 21:56:42.828430164 +0000 UTC m=+0.391221010 container start 7a2b6dbe5d8b26479ac240c59da735275ab236572bc449f90775450eb416b4f9 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-12T21:56:43.094 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:56:42 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[42649]: ts=2024-04-12T21:56:42.850Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-04-12T21:56:43.094 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:56:42 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[42649]: ts=2024-04-12T21:56:42.850Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-04-12T21:56:43.094 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:56:42 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[42649]: ts=2024-04-12T21:56:42.850Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-04-12T21:56:43.095 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:56:42 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[42649]: ts=2024-04-12T21:56:42.850Z caller=main.go:518 level=info host_details="(Linux 5.14.0-435.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024 x86_64 smithi114 (none))" 2024-04-12T21:56:43.095 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:56:42 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[42649]: ts=2024-04-12T21:56:42.851Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-04-12T21:56:43.095 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:56:42 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[42649]: ts=2024-04-12T21:56:42.851Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-04-12T21:56:43.095 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:56:42 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[42649]: ts=2024-04-12T21:56:42.852Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-04-12T21:56:43.095 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:56:42 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[42649]: ts=2024-04-12T21:56:42.852Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-04-12T21:56:43.095 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:56:42 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[42649]: ts=2024-04-12T21:56:42.853Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-04-12T21:56:43.095 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:56:42 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[42649]: ts=2024-04-12T21:56:42.856Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-04-12T21:56:43.095 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:56:42 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[42649]: ts=2024-04-12T21:56:42.856Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.932µs 2024-04-12T21:56:43.095 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:56:42 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[42649]: ts=2024-04-12T21:56:42.856Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-04-12T21:56:43.095 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:56:42 smithi114 bash[42634]: 7a2b6dbe5d8b26479ac240c59da735275ab236572bc449f90775450eb416b4f9 2024-04-12T21:56:43.095 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:56:42 smithi114 systemd[1]: Started Ceph prometheus.a for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T21:56:43.095 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:56:43 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[42649]: ts=2024-04-12T21:56:43.092Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-04-12T21:56:43.095 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:56:43 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[42649]: ts=2024-04-12T21:56:43.092Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-04-12T21:56:43.096 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:56:43 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[42649]: ts=2024-04-12T21:56:43.092Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=23.721µs wal_replay_duration=236.52063ms total_replay_duration=236.557931ms 2024-04-12T21:56:43.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:42 smithi097 ceph-mon[24835]: osdmap e68: 8 total, 8 up, 8 in 2024-04-12T21:56:43.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:42 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/3870310708' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-04-12T21:56:43.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:42 smithi097 ceph-mon[24835]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-04-12T21:56:43.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:42 smithi097 ceph-mon[24835]: pgmap v63: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-12T21:56:43.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:42 smithi097 ceph-mon[29620]: osdmap e68: 8 total, 8 up, 8 in 2024-04-12T21:56:43.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:42 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/3870310708' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-04-12T21:56:43.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:42 smithi097 ceph-mon[29620]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-04-12T21:56:43.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:42 smithi097 ceph-mon[29620]: pgmap v63: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-12T21:56:43.494 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:56:43 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[42649]: ts=2024-04-12T21:56:43.095Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-04-12T21:56:43.494 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:56:43 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[42649]: ts=2024-04-12T21:56:43.095Z caller=main.go:947 level=info msg="TSDB started" 2024-04-12T21:56:43.494 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:56:43 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[42649]: ts=2024-04-12T21:56:43.095Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-04-12T21:56:43.494 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:56:43 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[42649]: ts=2024-04-12T21:56:43.112Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=16.791509ms db_storage=663ns remote_storage=1.206µs web_handler=309ns query_engine=667ns scrape=366.67µs scrape_sd=24.078µs notify=25.849µs notify_sd=7.045µs rules=16.14363ms 2024-04-12T21:56:43.494 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 21:56:43 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[42649]: ts=2024-04-12T21:56:43.112Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-04-12T21:56:44.246 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:43 smithi114 ceph-mon[25759]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-04-12T21:56:44.246 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:43 smithi114 ceph-mon[25759]: osdmap e69: 8 total, 8 up, 8 in 2024-04-12T21:56:44.246 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:43 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:44.246 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:43 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/3870310708' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-12T21:56:44.246 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:43 smithi114 ceph-mon[25759]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-12T21:56:44.247 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:43 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-12T21:56:44.247 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:43 smithi114 ceph-mon[25759]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-12T21:56:44.247 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:43 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.97:9093"}]: dispatch 2024-04-12T21:56:44.247 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:43 smithi114 ceph-mon[25759]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.97:9093"}]: dispatch 2024-04-12T21:56:44.247 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:43 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:44.247 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:43 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-12T21:56:44.247 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:43 smithi114 ceph-mon[25759]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-12T21:56:44.247 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:43 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.114:9095"}]: dispatch 2024-04-12T21:56:44.247 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:43 smithi114 ceph-mon[25759]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.114:9095"}]: dispatch 2024-04-12T21:56:44.247 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:43 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:44.247 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:43 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-12T21:56:44.247 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:43 smithi114 ceph-mon[25759]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-12T21:56:44.247 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:43 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.114:3000"}]: dispatch 2024-04-12T21:56:44.247 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:43 smithi114 ceph-mon[25759]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.114:3000"}]: dispatch 2024-04-12T21:56:44.248 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:43 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:44.248 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:43 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:56:44.248 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:43 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:56:44.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:43 smithi097 ceph-mon[29620]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-04-12T21:56:44.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:43 smithi097 ceph-mon[29620]: osdmap e69: 8 total, 8 up, 8 in 2024-04-12T21:56:44.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:43 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:44.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:43 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/3870310708' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-12T21:56:44.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:43 smithi097 ceph-mon[29620]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-12T21:56:44.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:43 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-12T21:56:44.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:43 smithi097 ceph-mon[29620]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-12T21:56:44.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:43 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.97:9093"}]: dispatch 2024-04-12T21:56:44.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:43 smithi097 ceph-mon[29620]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.97:9093"}]: dispatch 2024-04-12T21:56:44.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:43 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:44.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:43 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-12T21:56:44.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:43 smithi097 ceph-mon[29620]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-12T21:56:44.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:43 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.114:9095"}]: dispatch 2024-04-12T21:56:44.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:43 smithi097 ceph-mon[29620]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.114:9095"}]: dispatch 2024-04-12T21:56:44.413 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:43 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:44.413 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:43 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-12T21:56:44.413 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:43 smithi097 ceph-mon[29620]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-12T21:56:44.413 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:43 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.114:3000"}]: dispatch 2024-04-12T21:56:44.413 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:43 smithi097 ceph-mon[29620]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.114:3000"}]: dispatch 2024-04-12T21:56:44.413 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:43 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:44.413 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:43 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:56:44.413 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:43 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:56:44.413 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:43 smithi097 ceph-mon[24835]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-04-12T21:56:44.413 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:43 smithi097 ceph-mon[24835]: osdmap e69: 8 total, 8 up, 8 in 2024-04-12T21:56:44.413 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:43 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:44.413 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:43 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/3870310708' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-12T21:56:44.414 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:43 smithi097 ceph-mon[24835]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-12T21:56:44.414 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:43 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-12T21:56:44.414 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:43 smithi097 ceph-mon[24835]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-12T21:56:44.414 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:43 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.97:9093"}]: dispatch 2024-04-12T21:56:44.414 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:43 smithi097 ceph-mon[24835]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.97:9093"}]: dispatch 2024-04-12T21:56:44.414 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:43 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:44.414 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:43 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-12T21:56:44.414 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:43 smithi097 ceph-mon[24835]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-12T21:56:44.414 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:43 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.114:9095"}]: dispatch 2024-04-12T21:56:44.414 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:43 smithi097 ceph-mon[24835]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.114:9095"}]: dispatch 2024-04-12T21:56:44.414 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:43 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:44.414 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:43 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-12T21:56:44.414 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:43 smithi097 ceph-mon[24835]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-12T21:56:44.414 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:43 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.114:3000"}]: dispatch 2024-04-12T21:56:44.415 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:43 smithi097 ceph-mon[24835]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.114:3000"}]: dispatch 2024-04-12T21:56:44.415 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:43 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:44.415 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:43 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:56:44.415 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:43 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:56:45.031 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-12T21:56:45.031 INFO:teuthology.orchestra.run.smithi097.stdout: "id": "af400bcc-91f7-4bf0-b607-4230acf8d27e", 2024-04-12T21:56:45.031 INFO:teuthology.orchestra.run.smithi097.stdout: "epoch": 1, 2024-04-12T21:56:45.031 INFO:teuthology.orchestra.run.smithi097.stdout: "predecessor_uuid": "ece10a89-1cd6-493b-b580-36ca0603baa2", 2024-04-12T21:56:45.031 INFO:teuthology.orchestra.run.smithi097.stdout: "sync_status": [], 2024-04-12T21:56:45.032 INFO:teuthology.orchestra.run.smithi097.stdout: "period_map": { 2024-04-12T21:56:45.032 INFO:teuthology.orchestra.run.smithi097.stdout: "id": "af400bcc-91f7-4bf0-b607-4230acf8d27e", 2024-04-12T21:56:45.032 INFO:teuthology.orchestra.run.smithi097.stdout: "zonegroups": [ 2024-04-12T21:56:45.032 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-12T21:56:45.032 INFO:teuthology.orchestra.run.smithi097.stdout: "id": "3879b12e-8b81-4a17-9f55-c05bead7a73d", 2024-04-12T21:56:45.032 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "default", 2024-04-12T21:56:45.032 INFO:teuthology.orchestra.run.smithi097.stdout: "api_name": "default", 2024-04-12T21:56:45.032 INFO:teuthology.orchestra.run.smithi097.stdout: "is_master": "true", 2024-04-12T21:56:45.032 INFO:teuthology.orchestra.run.smithi097.stdout: "endpoints": [], 2024-04-12T21:56:45.032 INFO:teuthology.orchestra.run.smithi097.stdout: "hostnames": [], 2024-04-12T21:56:45.032 INFO:teuthology.orchestra.run.smithi097.stdout: "hostnames_s3website": [], 2024-04-12T21:56:45.032 INFO:teuthology.orchestra.run.smithi097.stdout: "master_zone": "d22b1b60-1034-4324-aec2-c7a1b5510ffa", 2024-04-12T21:56:45.032 INFO:teuthology.orchestra.run.smithi097.stdout: "zones": [ 2024-04-12T21:56:45.032 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-12T21:56:45.033 INFO:teuthology.orchestra.run.smithi097.stdout: "id": "d22b1b60-1034-4324-aec2-c7a1b5510ffa", 2024-04-12T21:56:45.033 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "z", 2024-04-12T21:56:45.033 INFO:teuthology.orchestra.run.smithi097.stdout: "endpoints": [], 2024-04-12T21:56:45.033 INFO:teuthology.orchestra.run.smithi097.stdout: "log_meta": "false", 2024-04-12T21:56:45.033 INFO:teuthology.orchestra.run.smithi097.stdout: "log_data": "false", 2024-04-12T21:56:45.033 INFO:teuthology.orchestra.run.smithi097.stdout: "bucket_index_max_shards": 11, 2024-04-12T21:56:45.033 INFO:teuthology.orchestra.run.smithi097.stdout: "read_only": "false", 2024-04-12T21:56:45.033 INFO:teuthology.orchestra.run.smithi097.stdout: "tier_type": "", 2024-04-12T21:56:45.033 INFO:teuthology.orchestra.run.smithi097.stdout: "sync_from_all": "true", 2024-04-12T21:56:45.033 INFO:teuthology.orchestra.run.smithi097.stdout: "sync_from": [], 2024-04-12T21:56:45.033 INFO:teuthology.orchestra.run.smithi097.stdout: "redirect_zone": "" 2024-04-12T21:56:45.033 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-12T21:56:45.033 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-04-12T21:56:45.033 INFO:teuthology.orchestra.run.smithi097.stdout: "placement_targets": [ 2024-04-12T21:56:45.033 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-12T21:56:45.034 INFO:teuthology.orchestra.run.smithi097.stdout: "name": "default-placement", 2024-04-12T21:56:45.034 INFO:teuthology.orchestra.run.smithi097.stdout: "tags": [], 2024-04-12T21:56:45.034 INFO:teuthology.orchestra.run.smithi097.stdout: "storage_classes": [ 2024-04-12T21:56:45.034 INFO:teuthology.orchestra.run.smithi097.stdout: "STANDARD" 2024-04-12T21:56:45.034 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-12T21:56:45.034 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-12T21:56:45.034 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-04-12T21:56:45.034 INFO:teuthology.orchestra.run.smithi097.stdout: "default_placement": "default-placement", 2024-04-12T21:56:45.034 INFO:teuthology.orchestra.run.smithi097.stdout: "realm_id": "8e06d417-cc67-4293-b394-c4ab3e41b979", 2024-04-12T21:56:45.034 INFO:teuthology.orchestra.run.smithi097.stdout: "sync_policy": { 2024-04-12T21:56:45.034 INFO:teuthology.orchestra.run.smithi097.stdout: "groups": [] 2024-04-12T21:56:45.034 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-12T21:56:45.034 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-12T21:56:45.035 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-04-12T21:56:45.035 INFO:teuthology.orchestra.run.smithi097.stdout: "short_zone_ids": [ 2024-04-12T21:56:45.035 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-04-12T21:56:45.035 INFO:teuthology.orchestra.run.smithi097.stdout: "key": "d22b1b60-1034-4324-aec2-c7a1b5510ffa", 2024-04-12T21:56:45.035 INFO:teuthology.orchestra.run.smithi097.stdout: "val": 1114539898 2024-04-12T21:56:45.035 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-12T21:56:45.035 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-04-12T21:56:45.035 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T21:56:45.035 INFO:teuthology.orchestra.run.smithi097.stdout: "master_zonegroup": "3879b12e-8b81-4a17-9f55-c05bead7a73d", 2024-04-12T21:56:45.035 INFO:teuthology.orchestra.run.smithi097.stdout: "master_zone": "d22b1b60-1034-4324-aec2-c7a1b5510ffa", 2024-04-12T21:56:45.035 INFO:teuthology.orchestra.run.smithi097.stdout: "period_config": { 2024-04-12T21:56:45.035 INFO:teuthology.orchestra.run.smithi097.stdout: "bucket_quota": { 2024-04-12T21:56:45.035 INFO:teuthology.orchestra.run.smithi097.stdout: "enabled": false, 2024-04-12T21:56:45.035 INFO:teuthology.orchestra.run.smithi097.stdout: "check_on_raw": false, 2024-04-12T21:56:45.035 INFO:teuthology.orchestra.run.smithi097.stdout: "max_size": -1, 2024-04-12T21:56:45.035 INFO:teuthology.orchestra.run.smithi097.stdout: "max_size_kb": 0, 2024-04-12T21:56:45.036 INFO:teuthology.orchestra.run.smithi097.stdout: "max_objects": -1 2024-04-12T21:56:45.036 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T21:56:45.036 INFO:teuthology.orchestra.run.smithi097.stdout: "user_quota": { 2024-04-12T21:56:45.036 INFO:teuthology.orchestra.run.smithi097.stdout: "enabled": false, 2024-04-12T21:56:45.036 INFO:teuthology.orchestra.run.smithi097.stdout: "check_on_raw": false, 2024-04-12T21:56:45.036 INFO:teuthology.orchestra.run.smithi097.stdout: "max_size": -1, 2024-04-12T21:56:45.036 INFO:teuthology.orchestra.run.smithi097.stdout: "max_size_kb": 0, 2024-04-12T21:56:45.036 INFO:teuthology.orchestra.run.smithi097.stdout: "max_objects": -1 2024-04-12T21:56:45.036 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T21:56:45.036 INFO:teuthology.orchestra.run.smithi097.stdout: "user_ratelimit": { 2024-04-12T21:56:45.036 INFO:teuthology.orchestra.run.smithi097.stdout: "max_read_ops": 0, 2024-04-12T21:56:45.036 INFO:teuthology.orchestra.run.smithi097.stdout: "max_write_ops": 0, 2024-04-12T21:56:45.036 INFO:teuthology.orchestra.run.smithi097.stdout: "max_read_bytes": 0, 2024-04-12T21:56:45.036 INFO:teuthology.orchestra.run.smithi097.stdout: "max_write_bytes": 0, 2024-04-12T21:56:45.036 INFO:teuthology.orchestra.run.smithi097.stdout: "enabled": false 2024-04-12T21:56:45.037 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T21:56:45.037 INFO:teuthology.orchestra.run.smithi097.stdout: "bucket_ratelimit": { 2024-04-12T21:56:45.037 INFO:teuthology.orchestra.run.smithi097.stdout: "max_read_ops": 0, 2024-04-12T21:56:45.037 INFO:teuthology.orchestra.run.smithi097.stdout: "max_write_ops": 0, 2024-04-12T21:56:45.037 INFO:teuthology.orchestra.run.smithi097.stdout: "max_read_bytes": 0, 2024-04-12T21:56:45.037 INFO:teuthology.orchestra.run.smithi097.stdout: "max_write_bytes": 0, 2024-04-12T21:56:45.037 INFO:teuthology.orchestra.run.smithi097.stdout: "enabled": false 2024-04-12T21:56:45.037 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T21:56:45.037 INFO:teuthology.orchestra.run.smithi097.stdout: "anonymous_ratelimit": { 2024-04-12T21:56:45.037 INFO:teuthology.orchestra.run.smithi097.stdout: "max_read_ops": 0, 2024-04-12T21:56:45.037 INFO:teuthology.orchestra.run.smithi097.stdout: "max_write_ops": 0, 2024-04-12T21:56:45.037 INFO:teuthology.orchestra.run.smithi097.stdout: "max_read_bytes": 0, 2024-04-12T21:56:45.037 INFO:teuthology.orchestra.run.smithi097.stdout: "max_write_bytes": 0, 2024-04-12T21:56:45.037 INFO:teuthology.orchestra.run.smithi097.stdout: "enabled": false 2024-04-12T21:56:45.038 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-12T21:56:45.038 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T21:56:45.038 INFO:teuthology.orchestra.run.smithi097.stdout: "realm_id": "8e06d417-cc67-4293-b394-c4ab3e41b979", 2024-04-12T21:56:45.038 INFO:teuthology.orchestra.run.smithi097.stdout: "realm_name": "r", 2024-04-12T21:56:45.038 INFO:teuthology.orchestra.run.smithi097.stdout: "realm_epoch": 2 2024-04-12T21:56:45.038 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-12T21:56:45.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:44 smithi114 ceph-mon[25759]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-12T21:56:45.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:44 smithi114 ceph-mon[25759]: osdmap e70: 8 total, 8 up, 8 in 2024-04-12T21:56:45.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:44 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/3870310708' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-04-12T21:56:45.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:44 smithi114 ceph-mon[25759]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-04-12T21:56:45.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:44 smithi114 ceph-mon[25759]: pgmap v66: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-12T21:56:45.341 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:44 smithi097 ceph-mon[24835]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-12T21:56:45.341 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:44 smithi097 ceph-mon[24835]: osdmap e70: 8 total, 8 up, 8 in 2024-04-12T21:56:45.341 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:44 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/3870310708' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-04-12T21:56:45.341 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:44 smithi097 ceph-mon[24835]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-04-12T21:56:45.341 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:44 smithi097 ceph-mon[24835]: pgmap v66: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-12T21:56:45.342 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:44 smithi097 ceph-mon[29620]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-12T21:56:45.342 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:44 smithi097 ceph-mon[29620]: osdmap e70: 8 total, 8 up, 8 in 2024-04-12T21:56:45.342 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:44 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/3870310708' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-04-12T21:56:45.342 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:44 smithi097 ceph-mon[29620]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-04-12T21:56:45.342 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:44 smithi097 ceph-mon[29620]: pgmap v66: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-12T21:56:45.809 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -e sha1=b277feb2c7beb4597d8ecc39d58d5c36b81f613a -- bash -c 'ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000' 2024-04-12T21:56:46.069 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:45 smithi097 ceph-mon[24835]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-04-12T21:56:46.069 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:45 smithi097 ceph-mon[24835]: osdmap e71: 8 total, 8 up, 8 in 2024-04-12T21:56:46.070 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:45 smithi097 ceph-mon[29620]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-04-12T21:56:46.070 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:45 smithi097 ceph-mon[29620]: osdmap e71: 8 total, 8 up, 8 in 2024-04-12T21:56:46.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:45 smithi114 ceph-mon[25759]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-04-12T21:56:46.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:45 smithi114 ceph-mon[25759]: osdmap e71: 8 total, 8 up, 8 in 2024-04-12T21:56:47.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:47 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:47.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:47 smithi097 ceph-mon[24835]: pgmap v68: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:56:47.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:47 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:47.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:47 smithi097 ceph-mon[29620]: pgmap v68: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:56:47.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:47 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:47.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:47 smithi114 ceph-mon[25759]: pgmap v68: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:56:48.473 INFO:teuthology.orchestra.run.smithi097.stdout:Scheduled rgw.foo update... 2024-04-12T21:56:48.473 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 21:56:48 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[49677]: level=info ts=2024-04-12T21:56:48.305Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.00192061s 2024-04-12T21:56:49.120 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -e sha1=b277feb2c7beb4597d8ecc39d58d5c36b81f613a -- bash -c 'ceph orch apply rgw smpl' 2024-04-12T21:56:49.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:49 smithi114 ceph-mon[25759]: from='client.24607 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:56:49.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:49 smithi114 ceph-mon[25759]: Saving service rgw.foo spec with placement count:2 2024-04-12T21:56:49.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:49 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:49.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:49 smithi114 ceph-mon[25759]: pgmap v69: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-12T21:56:49.772 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:49 smithi097 ceph-mon[24835]: from='client.24607 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:56:49.772 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:49 smithi097 ceph-mon[24835]: Saving service rgw.foo spec with placement count:2 2024-04-12T21:56:49.772 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:49 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:49.772 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:49 smithi097 ceph-mon[24835]: pgmap v69: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-12T21:56:49.772 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:49 smithi097 ceph-mon[29620]: from='client.24607 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:56:49.772 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:49 smithi097 ceph-mon[29620]: Saving service rgw.foo spec with placement count:2 2024-04-12T21:56:49.773 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:49 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:49.773 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:49 smithi097 ceph-mon[29620]: pgmap v69: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-12T21:56:51.234 INFO:teuthology.orchestra.run.smithi097.stdout:Scheduled rgw.smpl update... 2024-04-12T21:56:51.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:51 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:51.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:51 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:51.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:51 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:51.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:51 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:51.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:51 smithi097 ceph-mon[29620]: Saving service rgw.foo spec with placement count:2 2024-04-12T21:56:51.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:51 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:51.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:51 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi097.sggtkh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-12T21:56:51.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:51 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi097.sggtkh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-12T21:56:51.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:51 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi097.sggtkh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-12T21:56:51.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:51 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:51.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:51 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:56:51.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:51 smithi097 ceph-mon[29620]: Deploying daemon rgw.foo.smithi097.sggtkh on smithi097 2024-04-12T21:56:51.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:51 smithi097 ceph-mon[29620]: pgmap v70: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 937 B/s rd, 937 B/s wr, 2 op/s 2024-04-12T21:56:51.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:51 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:51.663 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:51 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:51.663 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:51 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:51.663 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:51 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:51.663 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:51 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:51.663 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:51 smithi097 ceph-mon[24835]: Saving service rgw.foo spec with placement count:2 2024-04-12T21:56:51.663 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:51 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:51.663 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:51 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi097.sggtkh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-12T21:56:51.663 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:51 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi097.sggtkh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-12T21:56:51.663 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:51 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi097.sggtkh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-12T21:56:51.663 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:51 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:51.663 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:51 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:56:51.663 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:51 smithi097 ceph-mon[24835]: Deploying daemon rgw.foo.smithi097.sggtkh on smithi097 2024-04-12T21:56:51.663 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:51 smithi097 ceph-mon[24835]: pgmap v70: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 937 B/s rd, 937 B/s wr, 2 op/s 2024-04-12T21:56:51.664 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:51 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:51.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:51 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:51.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:51 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:51.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:51 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:51.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:51 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:51.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:51 smithi114 ceph-mon[25759]: Saving service rgw.foo spec with placement count:2 2024-04-12T21:56:51.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:51 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:51.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:51 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi097.sggtkh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-12T21:56:51.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:51 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi097.sggtkh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-12T21:56:51.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:51 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi097.sggtkh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-12T21:56:51.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:51 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:51.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:51 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:56:51.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:51 smithi114 ceph-mon[25759]: Deploying daemon rgw.foo.smithi097.sggtkh on smithi097 2024-04-12T21:56:51.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:51 smithi114 ceph-mon[25759]: pgmap v70: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 937 B/s rd, 937 B/s wr, 2 op/s 2024-04-12T21:56:51.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:51 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:52.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:52 smithi097 ceph-mon[29620]: from='client.24613 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:56:52.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:52 smithi097 ceph-mon[29620]: Saving service rgw.smpl spec with placement count:2 2024-04-12T21:56:52.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:52 smithi097 ceph-mon[24835]: from='client.24613 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:56:52.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:52 smithi097 ceph-mon[24835]: Saving service rgw.smpl spec with placement count:2 2024-04-12T21:56:52.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:52 smithi114 ceph-mon[25759]: from='client.24613 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:56:52.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:52 smithi114 ceph-mon[25759]: Saving service rgw.smpl spec with placement count:2 2024-04-12T21:56:52.870 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -e sha1=b277feb2c7beb4597d8ecc39d58d5c36b81f613a -- bash -c 'ceph osd pool create foo' 2024-04-12T21:56:53.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:53 smithi097 ceph-mon[24835]: pgmap v71: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 829 B/s rd, 829 B/s wr, 2 op/s 2024-04-12T21:56:53.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:53 smithi097 ceph-mon[29620]: pgmap v71: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 829 B/s rd, 829 B/s wr, 2 op/s 2024-04-12T21:56:53.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:53 smithi114 ceph-mon[25759]: pgmap v71: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 829 B/s rd, 829 B/s wr, 2 op/s 2024-04-12T21:56:54.744 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:56:54 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: ::ffff:172.21.15.114 - - [12/Apr/2024:21:56:54] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-12T21:56:55.680 INFO:teuthology.orchestra.run.smithi097.stderr:pool 'foo' created 2024-04-12T21:56:55.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:55 smithi097 ceph-mon[24835]: pgmap v72: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 716 B/s wr, 1 op/s 2024-04-12T21:56:55.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:55 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/2893452302' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-04-12T21:56:55.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:55 smithi097 ceph-mon[29620]: pgmap v72: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 716 B/s wr, 1 op/s 2024-04-12T21:56:55.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:55 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/2893452302' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-04-12T21:56:55.938 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:55 smithi114 ceph-mon[25759]: pgmap v72: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 716 B/s wr, 1 op/s 2024-04-12T21:56:55.938 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:55 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/2893452302' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-04-12T21:56:56.411 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:56:56 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: ::ffff:172.21.15.114 - - [12/Apr/2024:21:56:56] "GET /metrics HTTP/1.1" 200 196724 "" "Prometheus/2.33.4" 2024-04-12T21:56:56.567 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -e sha1=b277feb2c7beb4597d8ecc39d58d5c36b81f613a -- bash -c 'rbd pool init foo' 2024-04-12T21:56:56.895 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:56 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/2893452302' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-04-12T21:56:56.896 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:56 smithi097 ceph-mon[29620]: osdmap e72: 8 total, 8 up, 8 in 2024-04-12T21:56:56.896 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:56 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:56.896 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:56 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi114.ulfbyo", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-12T21:56:56.896 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:56 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi114.ulfbyo", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-12T21:56:56.896 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:56 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi114.ulfbyo", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-12T21:56:56.896 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:56 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:56.896 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:56 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:56:56.896 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:56 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/2893452302' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-04-12T21:56:56.896 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:56 smithi097 ceph-mon[24835]: osdmap e72: 8 total, 8 up, 8 in 2024-04-12T21:56:56.896 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:56 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:56.896 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:56 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi114.ulfbyo", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-12T21:56:56.896 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:56 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi114.ulfbyo", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-12T21:56:56.896 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:56 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi114.ulfbyo", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-12T21:56:56.897 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:56 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:56.897 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:56 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:56:56.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:56 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/2893452302' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-04-12T21:56:56.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:56 smithi114 ceph-mon[25759]: osdmap e72: 8 total, 8 up, 8 in 2024-04-12T21:56:56.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:56 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:56.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:56 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi114.ulfbyo", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-12T21:56:56.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:56 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi114.ulfbyo", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-12T21:56:56.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:56 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi114.ulfbyo", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-12T21:56:56.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:56 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:56.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:56 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:56:57.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:57 smithi114 ceph-mon[25759]: Deploying daemon rgw.foo.smithi114.ulfbyo on smithi114 2024-04-12T21:56:57.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:57 smithi114 ceph-mon[25759]: pgmap v74: 161 pgs: 3 creating+peering, 129 active+clean, 29 unknown; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 7.9 KiB/s rd, 1.4 KiB/s wr, 18 op/s 2024-04-12T21:56:57.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:57 smithi114 ceph-mon[25759]: osdmap e73: 8 total, 8 up, 8 in 2024-04-12T21:56:57.929 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:57 smithi097 ceph-mon[29620]: Deploying daemon rgw.foo.smithi114.ulfbyo on smithi114 2024-04-12T21:56:57.929 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:57 smithi097 ceph-mon[29620]: pgmap v74: 161 pgs: 3 creating+peering, 129 active+clean, 29 unknown; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 7.9 KiB/s rd, 1.4 KiB/s wr, 18 op/s 2024-04-12T21:56:57.929 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:57 smithi097 ceph-mon[29620]: osdmap e73: 8 total, 8 up, 8 in 2024-04-12T21:56:57.930 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:57 smithi097 ceph-mon[24835]: Deploying daemon rgw.foo.smithi114.ulfbyo on smithi114 2024-04-12T21:56:57.930 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:57 smithi097 ceph-mon[24835]: pgmap v74: 161 pgs: 3 creating+peering, 129 active+clean, 29 unknown; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 7.9 KiB/s rd, 1.4 KiB/s wr, 18 op/s 2024-04-12T21:56:57.930 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:57 smithi097 ceph-mon[24835]: osdmap e73: 8 total, 8 up, 8 in 2024-04-12T21:56:58.882 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:58 smithi097 ceph-mon[24835]: osdmap e74: 8 total, 8 up, 8 in 2024-04-12T21:56:58.882 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:58 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:58.882 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:58 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:56:58.882 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:58 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:56:58.882 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:58 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/2390343632' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-04-12T21:56:58.882 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:58 smithi097 ceph-mon[24835]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-04-12T21:56:58.882 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:58 smithi097 ceph-mon[29620]: osdmap e74: 8 total, 8 up, 8 in 2024-04-12T21:56:58.883 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:58 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:58.883 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:58 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:56:58.883 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:58 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:56:58.883 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:58 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/2390343632' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-04-12T21:56:58.883 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:58 smithi097 ceph-mon[29620]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-04-12T21:56:58.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:58 smithi114 ceph-mon[25759]: osdmap e74: 8 total, 8 up, 8 in 2024-04-12T21:56:58.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:58 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:56:58.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:58 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:56:58.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:58 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:56:58.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:58 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/2390343632' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-04-12T21:56:58.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:58 smithi114 ceph-mon[25759]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-04-12T21:56:59.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:59 smithi114 ceph-mon[25759]: pgmap v77: 161 pgs: 32 creating+peering, 129 active+clean; 584 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 5.5 KiB/s wr, 186 op/s 2024-04-12T21:56:59.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:59 smithi114 ceph-mon[25759]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-04-12T21:56:59.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:56:59 smithi114 ceph-mon[25759]: osdmap e75: 8 total, 8 up, 8 in 2024-04-12T21:57:00.041 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:59 smithi097 ceph-mon[24835]: pgmap v77: 161 pgs: 32 creating+peering, 129 active+clean; 584 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 5.5 KiB/s wr, 186 op/s 2024-04-12T21:57:00.041 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:59 smithi097 ceph-mon[24835]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-04-12T21:57:00.041 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:56:59 smithi097 ceph-mon[24835]: osdmap e75: 8 total, 8 up, 8 in 2024-04-12T21:57:00.041 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:59 smithi097 ceph-mon[29620]: pgmap v77: 161 pgs: 32 creating+peering, 129 active+clean; 584 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 5.5 KiB/s wr, 186 op/s 2024-04-12T21:57:00.042 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:59 smithi097 ceph-mon[29620]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-04-12T21:57:00.042 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:56:59 smithi097 ceph-mon[29620]: osdmap e75: 8 total, 8 up, 8 in 2024-04-12T21:57:01.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:01 smithi097 ceph-mon[24835]: osdmap e76: 8 total, 8 up, 8 in 2024-04-12T21:57:01.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:01 smithi097 ceph-mon[24835]: pgmap v80: 161 pgs: 32 creating+peering, 129 active+clean; 584 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 98 KiB/s rd, 6.5 KiB/s wr, 237 op/s 2024-04-12T21:57:01.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:01 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T21:57:01.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:01 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T21:57:01.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:01 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T21:57:01.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:01 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:01.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:01 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T21:57:01.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:01 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:01.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:01 smithi097 ceph-mon[29620]: osdmap e76: 8 total, 8 up, 8 in 2024-04-12T21:57:01.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:01 smithi097 ceph-mon[29620]: pgmap v80: 161 pgs: 32 creating+peering, 129 active+clean; 584 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 98 KiB/s rd, 6.5 KiB/s wr, 237 op/s 2024-04-12T21:57:01.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:01 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T21:57:01.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:01 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T21:57:01.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:01 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T21:57:01.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:01 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:01.663 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:01 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T21:57:01.663 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:01 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:01.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:01 smithi114 ceph-mon[25759]: osdmap e76: 8 total, 8 up, 8 in 2024-04-12T21:57:01.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:01 smithi114 ceph-mon[25759]: pgmap v80: 161 pgs: 32 creating+peering, 129 active+clean; 584 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 98 KiB/s rd, 6.5 KiB/s wr, 237 op/s 2024-04-12T21:57:01.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:01 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T21:57:01.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:01 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T21:57:01.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:01 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T21:57:01.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:01 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:01.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:01 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T21:57:01.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:01 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:02.202 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -e sha1=b277feb2c7beb4597d8ecc39d58d5c36b81f613a -- bash -c 'ceph orch apply iscsi foo u p' 2024-04-12T21:57:02.449 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:02 smithi097 ceph-mon[24835]: osdmap e77: 8 total, 8 up, 8 in 2024-04-12T21:57:02.449 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:02 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:02.449 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:02 smithi097 ceph-mon[24835]: Checking dashboard <-> RGW credentials 2024-04-12T21:57:02.449 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:02 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:02.450 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:02 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:02.450 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:02 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:02.450 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:02 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:02.450 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:02 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi097.exxkgl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-12T21:57:02.450 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:02 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi097.exxkgl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-12T21:57:02.450 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:02 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi097.exxkgl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-12T21:57:02.450 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:02 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:02.450 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:02 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:57:02.450 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:02 smithi097 ceph-mon[29620]: osdmap e77: 8 total, 8 up, 8 in 2024-04-12T21:57:02.450 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:02 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:02.451 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:02 smithi097 ceph-mon[29620]: Checking dashboard <-> RGW credentials 2024-04-12T21:57:02.451 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:02 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:02.451 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:02 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:02.451 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:02 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:02.451 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:02 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:02.451 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:02 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi097.exxkgl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-12T21:57:02.451 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:02 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi097.exxkgl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-12T21:57:02.451 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:02 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi097.exxkgl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-12T21:57:02.451 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:02 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:02.451 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:02 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:57:02.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:02 smithi114 ceph-mon[25759]: osdmap e77: 8 total, 8 up, 8 in 2024-04-12T21:57:02.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:02 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:02.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:02 smithi114 ceph-mon[25759]: Checking dashboard <-> RGW credentials 2024-04-12T21:57:02.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:02 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:02.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:02 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:02.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:02 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:02.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:02 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:02.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:02 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi097.exxkgl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-12T21:57:02.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:02 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi097.exxkgl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-12T21:57:02.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:02 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi097.exxkgl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-12T21:57:02.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:02 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:02.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:02 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:57:03.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:03 smithi097 ceph-mon[24835]: Saving service rgw.smpl spec with placement count:2 2024-04-12T21:57:03.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:03 smithi097 ceph-mon[24835]: Deploying daemon rgw.smpl.smithi097.exxkgl on smithi097 2024-04-12T21:57:03.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:03 smithi097 ceph-mon[24835]: pgmap v82: 161 pgs: 161 active+clean; 584 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 133 KiB/s rd, 623 B/s wr, 205 op/s 2024-04-12T21:57:03.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:03 smithi097 ceph-mon[29620]: Saving service rgw.smpl spec with placement count:2 2024-04-12T21:57:03.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:03 smithi097 ceph-mon[29620]: Deploying daemon rgw.smpl.smithi097.exxkgl on smithi097 2024-04-12T21:57:03.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:03 smithi097 ceph-mon[29620]: pgmap v82: 161 pgs: 161 active+clean; 584 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 133 KiB/s rd, 623 B/s wr, 205 op/s 2024-04-12T21:57:03.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:03 smithi114 ceph-mon[25759]: Saving service rgw.smpl spec with placement count:2 2024-04-12T21:57:03.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:03 smithi114 ceph-mon[25759]: Deploying daemon rgw.smpl.smithi097.exxkgl on smithi097 2024-04-12T21:57:03.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:03 smithi114 ceph-mon[25759]: pgmap v82: 161 pgs: 161 active+clean; 584 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 133 KiB/s rd, 623 B/s wr, 205 op/s 2024-04-12T21:57:04.744 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:57:04 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: ::ffff:172.21.15.114 - - [12/Apr/2024:21:57:04] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-12T21:57:05.395 INFO:teuthology.orchestra.run.smithi097.stdout:Scheduled iscsi.foo update... 2024-04-12T21:57:05.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:05 smithi114 ceph-mon[25759]: pgmap v83: 161 pgs: 161 active+clean; 584 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 511 B/s wr, 168 op/s 2024-04-12T21:57:05.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:05 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:06.049 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:05 smithi097 ceph-mon[24835]: pgmap v83: 161 pgs: 161 active+clean; 584 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 511 B/s wr, 168 op/s 2024-04-12T21:57:06.050 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:05 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:06.050 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:05 smithi097 ceph-mon[29620]: pgmap v83: 161 pgs: 161 active+clean; 584 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 511 B/s wr, 168 op/s 2024-04-12T21:57:06.050 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:05 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:06.073 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -e sha1=b277feb2c7beb4597d8ecc39d58d5c36b81f613a -- bash -c 'sleep 120' 2024-04-12T21:57:06.316 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:57:06 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: ::ffff:172.21.15.114 - - [12/Apr/2024:21:57:06] "GET /metrics HTTP/1.1" 200 207094 "" "Prometheus/2.33.4" 2024-04-12T21:57:06.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:06 smithi097 ceph-mon[24835]: from='client.24709 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:57:06.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:06 smithi097 ceph-mon[24835]: Saving service iscsi.foo spec with placement count:1 2024-04-12T21:57:06.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:06 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:06.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:06 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi114.lrgvph", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-12T21:57:06.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:06 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi114.lrgvph", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-12T21:57:06.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:06 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi114.lrgvph", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-12T21:57:06.913 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:06 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:06.913 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:06 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:57:06.913 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:06 smithi097 ceph-mon[29620]: from='client.24709 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:57:06.913 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:06 smithi097 ceph-mon[29620]: Saving service iscsi.foo spec with placement count:1 2024-04-12T21:57:06.913 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:06 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:06.913 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:06 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi114.lrgvph", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-12T21:57:06.913 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:06 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi114.lrgvph", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-12T21:57:06.913 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:06 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi114.lrgvph", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-12T21:57:06.913 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:06 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:06.913 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:06 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:57:06.941 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:06 smithi114 ceph-mon[25759]: from='client.24709 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:57:06.941 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:06 smithi114 ceph-mon[25759]: Saving service iscsi.foo spec with placement count:1 2024-04-12T21:57:06.941 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:06 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:06.941 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:06 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi114.lrgvph", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-12T21:57:06.941 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:06 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi114.lrgvph", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-12T21:57:06.941 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:06 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi114.lrgvph", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-12T21:57:06.941 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:06 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:06.941 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:06 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:57:07.716 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:07 smithi097 ceph-mon[24835]: Deploying daemon rgw.smpl.smithi114.lrgvph on smithi114 2024-04-12T21:57:07.716 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:07 smithi097 ceph-mon[24835]: pgmap v84: 161 pgs: 161 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 168 KiB/s rd, 567 B/s wr, 257 op/s 2024-04-12T21:57:07.716 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:07 smithi097 ceph-mon[29620]: Deploying daemon rgw.smpl.smithi114.lrgvph on smithi114 2024-04-12T21:57:07.716 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:07 smithi097 ceph-mon[29620]: pgmap v84: 161 pgs: 161 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 168 KiB/s rd, 567 B/s wr, 257 op/s 2024-04-12T21:57:07.979 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:07 smithi114 ceph-mon[25759]: Deploying daemon rgw.smpl.smithi114.lrgvph on smithi114 2024-04-12T21:57:07.979 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:07 smithi114 ceph-mon[25759]: pgmap v84: 161 pgs: 161 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 168 KiB/s rd, 567 B/s wr, 257 op/s 2024-04-12T21:57:09.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:09 smithi114 ceph-mon[25759]: osdmap e78: 8 total, 8 up, 8 in 2024-04-12T21:57:09.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:09 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:09.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:09 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:57:09.495 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:09 smithi114 ceph-mon[25759]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 167 KiB/s rd, 767 B/s wr, 256 op/s 2024-04-12T21:57:09.495 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:09 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:57:09.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:09 smithi097 ceph-mon[24835]: osdmap e78: 8 total, 8 up, 8 in 2024-04-12T21:57:09.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:09 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:09.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:09 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:57:09.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:09 smithi097 ceph-mon[24835]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 167 KiB/s rd, 767 B/s wr, 256 op/s 2024-04-12T21:57:09.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:09 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:57:09.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:09 smithi097 ceph-mon[29620]: osdmap e78: 8 total, 8 up, 8 in 2024-04-12T21:57:09.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:09 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:09.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:09 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:57:09.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:09 smithi097 ceph-mon[29620]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 167 KiB/s rd, 767 B/s wr, 256 op/s 2024-04-12T21:57:09.663 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:09 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:57:11.677 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:11 smithi097 ceph-mon[24835]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 146 KiB/s rd, 667 B/s wr, 223 op/s 2024-04-12T21:57:11.677 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:11 smithi097 ceph-mon[29620]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 146 KiB/s rd, 667 B/s wr, 223 op/s 2024-04-12T21:57:11.677 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:11 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:11.677 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:11 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:11.962 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:11 smithi114 ceph-mon[25759]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 146 KiB/s rd, 667 B/s wr, 223 op/s 2024-04-12T21:57:11.962 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:11 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:11.962 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:11 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:12.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:11 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:12.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:11 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:13.525 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:13 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:13.525 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:13 smithi097 ceph-mon[29620]: Checking dashboard <-> RGW credentials 2024-04-12T21:57:13.525 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:13 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:13.525 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:13 smithi097 ceph-mon[29620]: Checking pool "foo" exists for service iscsi.foo 2024-04-12T21:57:13.525 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:13 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi097.xndkxy", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-12T21:57:13.525 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:13 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi097.xndkxy", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-12T21:57:13.526 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:13 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi097.xndkxy", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-04-12T21:57:13.526 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:13 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:57:13.526 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:13 smithi097 ceph-mon[29620]: Deploying daemon iscsi.foo.smithi097.xndkxy on smithi097 2024-04-12T21:57:13.526 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:13 smithi097 ceph-mon[29620]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 133 KiB/s rd, 409 B/s wr, 204 op/s 2024-04-12T21:57:13.526 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:13 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:13.526 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:13 smithi097 ceph-mon[24835]: Checking dashboard <-> RGW credentials 2024-04-12T21:57:13.526 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:13 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:13.526 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:13 smithi097 ceph-mon[24835]: Checking pool "foo" exists for service iscsi.foo 2024-04-12T21:57:13.527 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:13 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi097.xndkxy", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-12T21:57:13.527 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:13 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi097.xndkxy", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-12T21:57:13.527 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:13 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi097.xndkxy", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-04-12T21:57:13.527 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:13 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:57:13.527 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:13 smithi097 ceph-mon[24835]: Deploying daemon iscsi.foo.smithi097.xndkxy on smithi097 2024-04-12T21:57:13.527 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:13 smithi097 ceph-mon[24835]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 133 KiB/s rd, 409 B/s wr, 204 op/s 2024-04-12T21:57:13.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:13 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:13.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:13 smithi114 ceph-mon[25759]: Checking dashboard <-> RGW credentials 2024-04-12T21:57:13.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:13 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:13.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:13 smithi114 ceph-mon[25759]: Checking pool "foo" exists for service iscsi.foo 2024-04-12T21:57:13.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:13 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi097.xndkxy", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-12T21:57:13.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:13 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi097.xndkxy", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-12T21:57:13.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:13 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi097.xndkxy", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-04-12T21:57:13.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:13 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:57:13.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:13 smithi114 ceph-mon[25759]: Deploying daemon iscsi.foo.smithi097.xndkxy on smithi097 2024-04-12T21:57:13.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:13 smithi114 ceph-mon[25759]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 133 KiB/s rd, 409 B/s wr, 204 op/s 2024-04-12T21:57:13.911 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 21:57:13 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[49677]: level=warn ts=2024-04-12T21:57:13.522Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.114:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.114 because it doesn't contain any IP SANs" 2024-04-12T21:57:13.911 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 21:57:13 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[49677]: level=warn ts=2024-04-12T21:57:13.522Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.97:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.97 because it doesn't contain any IP SANs" 2024-04-12T21:57:14.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:14 smithi114 ceph-mon[25759]: mgrmap e20: y(active, since 2m), standbys: x 2024-04-12T21:57:14.744 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:57:14 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: ::ffff:172.21.15.114 - - [12/Apr/2024:21:57:14] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-12T21:57:14.764 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:14 smithi097 ceph-mon[24835]: mgrmap e20: y(active, since 2m), standbys: x 2024-04-12T21:57:14.764 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:14 smithi097 ceph-mon[29620]: mgrmap e20: y(active, since 2m), standbys: x 2024-04-12T21:57:15.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:15 smithi114 ceph-mon[25759]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 133 KiB/s rd, 409 B/s wr, 204 op/s 2024-04-12T21:57:15.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:15 smithi097 ceph-mon[24835]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 133 KiB/s rd, 409 B/s wr, 204 op/s 2024-04-12T21:57:15.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:15 smithi097 ceph-mon[29620]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 133 KiB/s rd, 409 B/s wr, 204 op/s 2024-04-12T21:57:16.211 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:57:16 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: ::ffff:172.21.15.114 - - [12/Apr/2024:21:57:16] "GET /metrics HTTP/1.1" 200 214881 "" "Prometheus/2.33.4" 2024-04-12T21:57:16.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:16 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:16.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:16 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:57:16.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:16 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:57:16.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:16 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:16.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:16 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/3832548503' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-12T21:57:16.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:16 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/1851408490' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/2357886921"}]: dispatch 2024-04-12T21:57:16.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:16 smithi097 ceph-mon[29620]: from='client.? ' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/2357886921"}]: dispatch 2024-04-12T21:57:16.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:16 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:16.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:16 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:57:16.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:16 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:57:16.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:16 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:16.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:16 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/3832548503' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-12T21:57:16.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:16 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/1851408490' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/2357886921"}]: dispatch 2024-04-12T21:57:16.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:16 smithi097 ceph-mon[24835]: from='client.? ' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/2357886921"}]: dispatch 2024-04-12T21:57:16.984 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:16 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:16.984 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:16 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:57:16.984 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:16 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:57:16.984 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:16 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:16.984 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:16 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/3832548503' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-12T21:57:16.984 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:16 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/1851408490' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/2357886921"}]: dispatch 2024-04-12T21:57:16.984 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:16 smithi114 ceph-mon[25759]: from='client.? ' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/2357886921"}]: dispatch 2024-04-12T21:57:17.960 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:17 smithi097 ceph-mon[29620]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 409 B/s wr, 124 op/s 2024-04-12T21:57:17.960 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:17 smithi097 ceph-mon[29620]: from='client.? ' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/2357886921"}]': finished 2024-04-12T21:57:17.960 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:17 smithi097 ceph-mon[29620]: osdmap e79: 8 total, 8 up, 8 in 2024-04-12T21:57:17.960 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:17 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/3288956289' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6801/2093806388"}]: dispatch 2024-04-12T21:57:17.960 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:17 smithi097 ceph-mon[24835]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 409 B/s wr, 124 op/s 2024-04-12T21:57:17.960 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:17 smithi097 ceph-mon[24835]: from='client.? ' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/2357886921"}]': finished 2024-04-12T21:57:17.960 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:17 smithi097 ceph-mon[24835]: osdmap e79: 8 total, 8 up, 8 in 2024-04-12T21:57:17.961 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:17 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/3288956289' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6801/2093806388"}]: dispatch 2024-04-12T21:57:17.964 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:17 smithi114 ceph-mon[25759]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 409 B/s wr, 124 op/s 2024-04-12T21:57:17.964 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:17 smithi114 ceph-mon[25759]: from='client.? ' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/2357886921"}]': finished 2024-04-12T21:57:17.964 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:17 smithi114 ceph-mon[25759]: osdmap e79: 8 total, 8 up, 8 in 2024-04-12T21:57:17.964 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:17 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/3288956289' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6801/2093806388"}]: dispatch 2024-04-12T21:57:18.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:18 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/3288956289' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6801/2093806388"}]': finished 2024-04-12T21:57:18.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:18 smithi097 ceph-mon[24835]: osdmap e80: 8 total, 8 up, 8 in 2024-04-12T21:57:18.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:18 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/323771049' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6800/2093806388"}]: dispatch 2024-04-12T21:57:18.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:18 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/3288956289' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6801/2093806388"}]': finished 2024-04-12T21:57:18.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:18 smithi097 ceph-mon[29620]: osdmap e80: 8 total, 8 up, 8 in 2024-04-12T21:57:18.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:18 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/323771049' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6800/2093806388"}]: dispatch 2024-04-12T21:57:18.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:18 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/3288956289' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6801/2093806388"}]': finished 2024-04-12T21:57:18.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:18 smithi114 ceph-mon[25759]: osdmap e80: 8 total, 8 up, 8 in 2024-04-12T21:57:18.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:18 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/323771049' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6800/2093806388"}]: dispatch 2024-04-12T21:57:19.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:19 smithi114 ceph-mon[25759]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 255 B/s wr, 131 op/s 2024-04-12T21:57:19.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:19 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/323771049' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6800/2093806388"}]': finished 2024-04-12T21:57:19.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:19 smithi114 ceph-mon[25759]: osdmap e81: 8 total, 8 up, 8 in 2024-04-12T21:57:19.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:19 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:19.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:19 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/3710787897' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/2789980025"}]: dispatch 2024-04-12T21:57:19.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:19 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:19.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:19 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:19.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:19 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-12T21:57:19.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:19 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-12T21:57:19.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:19 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:19.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:19 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi097"}]: dispatch 2024-04-12T21:57:19.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:19 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:19.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:19 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:57:19.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:19 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:57:19.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:19 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:20.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:19 smithi097 ceph-mon[24835]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 255 B/s wr, 131 op/s 2024-04-12T21:57:20.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:19 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/323771049' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6800/2093806388"}]': finished 2024-04-12T21:57:20.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:19 smithi097 ceph-mon[24835]: osdmap e81: 8 total, 8 up, 8 in 2024-04-12T21:57:20.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:19 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:20.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:19 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/3710787897' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/2789980025"}]: dispatch 2024-04-12T21:57:20.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:19 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:20.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:19 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:20.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:19 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-12T21:57:20.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:19 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-12T21:57:20.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:19 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:20.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:19 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi097"}]: dispatch 2024-04-12T21:57:20.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:19 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:20.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:19 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:57:20.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:19 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:57:20.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:19 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:20.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:19 smithi097 ceph-mon[29620]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 255 B/s wr, 131 op/s 2024-04-12T21:57:20.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:19 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/323771049' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6800/2093806388"}]': finished 2024-04-12T21:57:20.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:19 smithi097 ceph-mon[29620]: osdmap e81: 8 total, 8 up, 8 in 2024-04-12T21:57:20.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:19 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:20.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:19 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/3710787897' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/2789980025"}]: dispatch 2024-04-12T21:57:20.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:19 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:20.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:19 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:20.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:19 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-12T21:57:20.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:19 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-12T21:57:20.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:19 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:20.164 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:19 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi097"}]: dispatch 2024-04-12T21:57:20.164 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:19 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:20.164 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:19 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:57:20.164 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:19 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:57:20.164 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:19 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:57:20.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:20 smithi114 ceph-mon[25759]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-12T21:57:20.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:20 smithi114 ceph-mon[25759]: Adding iSCSI gateway http://:@172.21.15.97:5000 to Dashboard 2024-04-12T21:57:20.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:20 smithi114 ceph-mon[25759]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-12T21:57:20.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:20 smithi114 ceph-mon[25759]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi097"}]: dispatch 2024-04-12T21:57:20.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:20 smithi114 ceph-mon[25759]: Checking dashboard <-> RGW credentials 2024-04-12T21:57:20.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:20 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/3710787897' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/2789980025"}]': finished 2024-04-12T21:57:20.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:20 smithi114 ceph-mon[25759]: osdmap e82: 8 total, 8 up, 8 in 2024-04-12T21:57:20.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:20 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/1947389430' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/1481749366"}]: dispatch 2024-04-12T21:57:21.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:20 smithi097 ceph-mon[24835]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-12T21:57:21.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:20 smithi097 ceph-mon[24835]: Adding iSCSI gateway http://:@172.21.15.97:5000 to Dashboard 2024-04-12T21:57:21.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:20 smithi097 ceph-mon[24835]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-12T21:57:21.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:20 smithi097 ceph-mon[24835]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi097"}]: dispatch 2024-04-12T21:57:21.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:20 smithi097 ceph-mon[24835]: Checking dashboard <-> RGW credentials 2024-04-12T21:57:21.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:20 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/3710787897' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/2789980025"}]': finished 2024-04-12T21:57:21.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:20 smithi097 ceph-mon[24835]: osdmap e82: 8 total, 8 up, 8 in 2024-04-12T21:57:21.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:20 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/1947389430' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/1481749366"}]: dispatch 2024-04-12T21:57:21.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:20 smithi097 ceph-mon[29620]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-12T21:57:21.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:20 smithi097 ceph-mon[29620]: Adding iSCSI gateway http://:@172.21.15.97:5000 to Dashboard 2024-04-12T21:57:21.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:20 smithi097 ceph-mon[29620]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-12T21:57:21.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:20 smithi097 ceph-mon[29620]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi097"}]: dispatch 2024-04-12T21:57:21.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:20 smithi097 ceph-mon[29620]: Checking dashboard <-> RGW credentials 2024-04-12T21:57:21.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:20 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/3710787897' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/2789980025"}]': finished 2024-04-12T21:57:21.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:20 smithi097 ceph-mon[29620]: osdmap e82: 8 total, 8 up, 8 in 2024-04-12T21:57:21.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:20 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/1947389430' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/1481749366"}]: dispatch 2024-04-12T21:57:21.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:21 smithi114 ceph-mon[25759]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:57:21.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:21 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/1947389430' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/1481749366"}]': finished 2024-04-12T21:57:21.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:21 smithi114 ceph-mon[25759]: osdmap e83: 8 total, 8 up, 8 in 2024-04-12T21:57:21.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:21 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/2785586874' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/831596468"}]: dispatch 2024-04-12T21:57:22.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:21 smithi097 ceph-mon[29620]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:57:22.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:21 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/1947389430' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/1481749366"}]': finished 2024-04-12T21:57:22.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:21 smithi097 ceph-mon[29620]: osdmap e83: 8 total, 8 up, 8 in 2024-04-12T21:57:22.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:21 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/2785586874' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/831596468"}]: dispatch 2024-04-12T21:57:22.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:21 smithi097 ceph-mon[24835]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:57:22.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:21 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/1947389430' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/1481749366"}]': finished 2024-04-12T21:57:22.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:21 smithi097 ceph-mon[24835]: osdmap e83: 8 total, 8 up, 8 in 2024-04-12T21:57:22.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:21 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/2785586874' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/831596468"}]: dispatch 2024-04-12T21:57:23.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:22 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/2785586874' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/831596468"}]': finished 2024-04-12T21:57:23.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:22 smithi114 ceph-mon[25759]: osdmap e84: 8 total, 8 up, 8 in 2024-04-12T21:57:23.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:22 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/3876816758' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/1510317264"}]: dispatch 2024-04-12T21:57:23.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:22 smithi114 ceph-mon[25759]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-04-12T21:57:23.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:22 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/2785586874' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/831596468"}]': finished 2024-04-12T21:57:23.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:22 smithi097 ceph-mon[24835]: osdmap e84: 8 total, 8 up, 8 in 2024-04-12T21:57:23.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:22 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/3876816758' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/1510317264"}]: dispatch 2024-04-12T21:57:23.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:22 smithi097 ceph-mon[24835]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-04-12T21:57:23.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:22 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/2785586874' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/831596468"}]': finished 2024-04-12T21:57:23.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:22 smithi097 ceph-mon[29620]: osdmap e84: 8 total, 8 up, 8 in 2024-04-12T21:57:23.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:22 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/3876816758' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/1510317264"}]: dispatch 2024-04-12T21:57:23.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:22 smithi097 ceph-mon[29620]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-04-12T21:57:23.911 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 21:57:23 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[49677]: level=error ts=2024-04-12T21:57:23.508Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.97:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.97 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.114:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.114 because it doesn't contain any IP SANs" 2024-04-12T21:57:24.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:23 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/3876816758' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/1510317264"}]': finished 2024-04-12T21:57:24.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:23 smithi114 ceph-mon[25759]: osdmap e85: 8 total, 8 up, 8 in 2024-04-12T21:57:24.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:23 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/17257767' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/1562080679"}]: dispatch 2024-04-12T21:57:24.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:23 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/3876816758' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/1510317264"}]': finished 2024-04-12T21:57:24.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:23 smithi097 ceph-mon[24835]: osdmap e85: 8 total, 8 up, 8 in 2024-04-12T21:57:24.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:23 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/17257767' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/1562080679"}]: dispatch 2024-04-12T21:57:24.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:23 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/3876816758' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/1510317264"}]': finished 2024-04-12T21:57:24.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:23 smithi097 ceph-mon[29620]: osdmap e85: 8 total, 8 up, 8 in 2024-04-12T21:57:24.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:23 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/17257767' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/1562080679"}]: dispatch 2024-04-12T21:57:24.744 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:57:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: ::ffff:172.21.15.114 - - [12/Apr/2024:21:57:24] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-12T21:57:25.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:24 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/17257767' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/1562080679"}]': finished 2024-04-12T21:57:25.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:24 smithi114 ceph-mon[25759]: osdmap e86: 8 total, 8 up, 8 in 2024-04-12T21:57:25.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:24 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/1244979042' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6801/258588252"}]: dispatch 2024-04-12T21:57:25.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:24 smithi114 ceph-mon[25759]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-04-12T21:57:25.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:24 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/17257767' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/1562080679"}]': finished 2024-04-12T21:57:25.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:24 smithi097 ceph-mon[24835]: osdmap e86: 8 total, 8 up, 8 in 2024-04-12T21:57:25.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:24 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/1244979042' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6801/258588252"}]: dispatch 2024-04-12T21:57:25.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:24 smithi097 ceph-mon[24835]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-04-12T21:57:25.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:24 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/17257767' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/1562080679"}]': finished 2024-04-12T21:57:25.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:24 smithi097 ceph-mon[29620]: osdmap e86: 8 total, 8 up, 8 in 2024-04-12T21:57:25.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:24 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/1244979042' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6801/258588252"}]: dispatch 2024-04-12T21:57:25.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:24 smithi097 ceph-mon[29620]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-04-12T21:57:26.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:25 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/1244979042' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6801/258588252"}]': finished 2024-04-12T21:57:26.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:25 smithi114 ceph-mon[25759]: osdmap e87: 8 total, 8 up, 8 in 2024-04-12T21:57:26.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:25 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/4201455003' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/2696421935"}]: dispatch 2024-04-12T21:57:26.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:25 smithi114 ceph-mon[25759]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T21:57:26.411 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:57:26 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: ::ffff:172.21.15.114 - - [12/Apr/2024:21:57:26] "GET /metrics HTTP/1.1" 200 214881 "" "Prometheus/2.33.4" 2024-04-12T21:57:26.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:25 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/1244979042' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6801/258588252"}]': finished 2024-04-12T21:57:26.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:25 smithi097 ceph-mon[24835]: osdmap e87: 8 total, 8 up, 8 in 2024-04-12T21:57:26.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:25 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/4201455003' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/2696421935"}]: dispatch 2024-04-12T21:57:26.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:25 smithi097 ceph-mon[24835]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T21:57:26.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:25 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/1244979042' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6801/258588252"}]': finished 2024-04-12T21:57:26.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:25 smithi097 ceph-mon[29620]: osdmap e87: 8 total, 8 up, 8 in 2024-04-12T21:57:26.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:25 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/4201455003' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/2696421935"}]: dispatch 2024-04-12T21:57:26.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:25 smithi097 ceph-mon[29620]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T21:57:27.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:26 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/4201455003' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/2696421935"}]': finished 2024-04-12T21:57:27.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:26 smithi114 ceph-mon[25759]: osdmap e88: 8 total, 8 up, 8 in 2024-04-12T21:57:27.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:26 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/1622094463' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6800/258588252"}]: dispatch 2024-04-12T21:57:27.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:26 smithi114 ceph-mon[25759]: from='client.? ' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6800/258588252"}]: dispatch 2024-04-12T21:57:27.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:26 smithi114 ceph-mon[25759]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:57:27.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:26 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/4201455003' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/2696421935"}]': finished 2024-04-12T21:57:27.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:26 smithi097 ceph-mon[24835]: osdmap e88: 8 total, 8 up, 8 in 2024-04-12T21:57:27.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:26 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/1622094463' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6800/258588252"}]: dispatch 2024-04-12T21:57:27.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:26 smithi097 ceph-mon[24835]: from='client.? ' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6800/258588252"}]: dispatch 2024-04-12T21:57:27.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:26 smithi097 ceph-mon[24835]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:57:27.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:26 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/4201455003' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/2696421935"}]': finished 2024-04-12T21:57:27.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:26 smithi097 ceph-mon[29620]: osdmap e88: 8 total, 8 up, 8 in 2024-04-12T21:57:27.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:26 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/1622094463' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6800/258588252"}]: dispatch 2024-04-12T21:57:27.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:26 smithi097 ceph-mon[29620]: from='client.? ' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6800/258588252"}]: dispatch 2024-04-12T21:57:27.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:26 smithi097 ceph-mon[29620]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:57:28.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:27 smithi114 ceph-mon[25759]: from='client.? ' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6800/258588252"}]': finished 2024-04-12T21:57:28.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:27 smithi114 ceph-mon[25759]: osdmap e89: 8 total, 8 up, 8 in 2024-04-12T21:57:28.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:27 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/2467914512' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/2460245404"}]: dispatch 2024-04-12T21:57:28.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:27 smithi097 ceph-mon[29620]: from='client.? ' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6800/258588252"}]': finished 2024-04-12T21:57:28.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:27 smithi097 ceph-mon[29620]: osdmap e89: 8 total, 8 up, 8 in 2024-04-12T21:57:28.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:27 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/2467914512' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/2460245404"}]: dispatch 2024-04-12T21:57:28.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:27 smithi097 ceph-mon[24835]: from='client.? ' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6800/258588252"}]': finished 2024-04-12T21:57:28.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:27 smithi097 ceph-mon[24835]: osdmap e89: 8 total, 8 up, 8 in 2024-04-12T21:57:28.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:27 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/2467914512' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/2460245404"}]: dispatch 2024-04-12T21:57:29.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:28 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/2467914512' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/2460245404"}]': finished 2024-04-12T21:57:29.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:28 smithi114 ceph-mon[25759]: osdmap e90: 8 total, 8 up, 8 in 2024-04-12T21:57:29.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:28 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/1563453227' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/1828347223"}]: dispatch 2024-04-12T21:57:29.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:28 smithi114 ceph-mon[25759]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:57:29.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:28 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/2467914512' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/2460245404"}]': finished 2024-04-12T21:57:29.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:28 smithi097 ceph-mon[24835]: osdmap e90: 8 total, 8 up, 8 in 2024-04-12T21:57:29.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:28 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/1563453227' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/1828347223"}]: dispatch 2024-04-12T21:57:29.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:28 smithi097 ceph-mon[24835]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:57:29.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:28 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/2467914512' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/2460245404"}]': finished 2024-04-12T21:57:29.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:28 smithi097 ceph-mon[29620]: osdmap e90: 8 total, 8 up, 8 in 2024-04-12T21:57:29.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:28 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/1563453227' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/1828347223"}]: dispatch 2024-04-12T21:57:29.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:28 smithi097 ceph-mon[29620]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:57:30.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:29 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/1563453227' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/1828347223"}]': finished 2024-04-12T21:57:30.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:29 smithi114 ceph-mon[25759]: osdmap e91: 8 total, 8 up, 8 in 2024-04-12T21:57:30.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:29 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/402885798' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6800/1796168157"}]: dispatch 2024-04-12T21:57:30.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:29 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/1563453227' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/1828347223"}]': finished 2024-04-12T21:57:30.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:29 smithi097 ceph-mon[24835]: osdmap e91: 8 total, 8 up, 8 in 2024-04-12T21:57:30.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:29 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/402885798' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6800/1796168157"}]: dispatch 2024-04-12T21:57:30.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:29 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/1563453227' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/1828347223"}]': finished 2024-04-12T21:57:30.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:29 smithi097 ceph-mon[29620]: osdmap e91: 8 total, 8 up, 8 in 2024-04-12T21:57:30.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:29 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/402885798' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6800/1796168157"}]: dispatch 2024-04-12T21:57:31.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:30 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/402885798' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6800/1796168157"}]': finished 2024-04-12T21:57:31.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:30 smithi114 ceph-mon[25759]: osdmap e92: 8 total, 8 up, 8 in 2024-04-12T21:57:31.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:30 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/3030435802' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6801/1796168157"}]: dispatch 2024-04-12T21:57:31.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:30 smithi114 ceph-mon[25759]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:57:31.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:30 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/402885798' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6800/1796168157"}]': finished 2024-04-12T21:57:31.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:30 smithi097 ceph-mon[24835]: osdmap e92: 8 total, 8 up, 8 in 2024-04-12T21:57:31.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:30 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/3030435802' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6801/1796168157"}]: dispatch 2024-04-12T21:57:31.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:30 smithi097 ceph-mon[24835]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:57:31.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:30 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/402885798' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6800/1796168157"}]': finished 2024-04-12T21:57:31.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:30 smithi097 ceph-mon[29620]: osdmap e92: 8 total, 8 up, 8 in 2024-04-12T21:57:31.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:30 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/3030435802' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6801/1796168157"}]: dispatch 2024-04-12T21:57:31.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:30 smithi097 ceph-mon[29620]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:57:32.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:31 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/3030435802' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6801/1796168157"}]': finished 2024-04-12T21:57:32.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:31 smithi114 ceph-mon[25759]: osdmap e93: 8 total, 8 up, 8 in 2024-04-12T21:57:32.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:31 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/398769595' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/1917553394"}]: dispatch 2024-04-12T21:57:32.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:31 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/3030435802' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6801/1796168157"}]': finished 2024-04-12T21:57:32.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:31 smithi097 ceph-mon[24835]: osdmap e93: 8 total, 8 up, 8 in 2024-04-12T21:57:32.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:31 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/398769595' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/1917553394"}]: dispatch 2024-04-12T21:57:32.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:31 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/3030435802' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6801/1796168157"}]': finished 2024-04-12T21:57:32.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:31 smithi097 ceph-mon[29620]: osdmap e93: 8 total, 8 up, 8 in 2024-04-12T21:57:32.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:31 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/398769595' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/1917553394"}]: dispatch 2024-04-12T21:57:33.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:32 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/398769595' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/1917553394"}]': finished 2024-04-12T21:57:33.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:32 smithi097 ceph-mon[24835]: osdmap e94: 8 total, 8 up, 8 in 2024-04-12T21:57:33.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:32 smithi097 ceph-mon[24835]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:57:33.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:32 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/398769595' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/1917553394"}]': finished 2024-04-12T21:57:33.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:32 smithi097 ceph-mon[29620]: osdmap e94: 8 total, 8 up, 8 in 2024-04-12T21:57:33.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:32 smithi097 ceph-mon[29620]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:57:33.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:32 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/398769595' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/1917553394"}]': finished 2024-04-12T21:57:33.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:32 smithi114 ceph-mon[25759]: osdmap e94: 8 total, 8 up, 8 in 2024-04-12T21:57:33.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:32 smithi114 ceph-mon[25759]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:57:34.744 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:57:34 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: ::ffff:172.21.15.114 - - [12/Apr/2024:21:57:34] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-12T21:57:35.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:35 smithi097 ceph-mon[24835]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 904 B/s rd, 0 op/s 2024-04-12T21:57:35.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:35 smithi097 ceph-mon[29620]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 904 B/s rd, 0 op/s 2024-04-12T21:57:35.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:35 smithi114 ceph-mon[25759]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 904 B/s rd, 0 op/s 2024-04-12T21:57:36.661 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:57:36 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: ::ffff:172.21.15.114 - - [12/Apr/2024:21:57:36] "GET /metrics HTTP/1.1" 200 214913 "" "Prometheus/2.33.4" 2024-04-12T21:57:36.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:36 smithi114 ceph-mon[25759]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T21:57:37.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:36 smithi097 ceph-mon[24835]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T21:57:37.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:36 smithi097 ceph-mon[29620]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T21:57:37.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:37 smithi114 ceph-mon[25759]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-12T21:57:38.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:37 smithi097 ceph-mon[24835]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-12T21:57:38.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:37 smithi097 ceph-mon[29620]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-12T21:57:39.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:39 smithi097 ceph-mon[24835]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:57:39.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:39 smithi097 ceph-mon[29620]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:57:39.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:39 smithi114 ceph-mon[25759]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:57:41.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:41 smithi114 ceph-mon[25759]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-12T21:57:42.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:41 smithi097 ceph-mon[24835]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-12T21:57:42.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:41 smithi097 ceph-mon[29620]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-12T21:57:43.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:43 smithi114 ceph-mon[25759]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 963 B/s rd, 0 op/s 2024-04-12T21:57:44.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:43 smithi097 ceph-mon[24835]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 963 B/s rd, 0 op/s 2024-04-12T21:57:44.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:43 smithi097 ceph-mon[29620]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 963 B/s rd, 0 op/s 2024-04-12T21:57:44.744 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:57:44 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: ::ffff:172.21.15.114 - - [12/Apr/2024:21:57:44] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-12T21:57:45.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:45 smithi114 ceph-mon[25759]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:57:46.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:45 smithi097 ceph-mon[24835]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:57:46.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:45 smithi097 ceph-mon[29620]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:57:46.661 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:57:46 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: ::ffff:172.21.15.114 - - [12/Apr/2024:21:57:46] "GET /metrics HTTP/1.1" 200 214836 "" "Prometheus/2.33.4" 2024-04-12T21:57:46.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:46 smithi114 ceph-mon[25759]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T21:57:47.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:46 smithi097 ceph-mon[24835]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T21:57:47.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:46 smithi097 ceph-mon[29620]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T21:57:47.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:47 smithi114 ceph-mon[25759]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:57:48.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:47 smithi097 ceph-mon[24835]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:57:48.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:47 smithi097 ceph-mon[29620]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:57:49.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:49 smithi097 ceph-mon[24835]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:57:49.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:49 smithi097 ceph-mon[29620]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:57:49.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:49 smithi114 ceph-mon[25759]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:57:51.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:51 smithi114 ceph-mon[25759]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:57:52.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:51 smithi097 ceph-mon[24835]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:57:52.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:51 smithi097 ceph-mon[29620]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:57:53.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:53 smithi114 ceph-mon[25759]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:57:54.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:53 smithi097 ceph-mon[24835]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:57:54.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:53 smithi097 ceph-mon[29620]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:57:54.744 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:57:54 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: ::ffff:172.21.15.114 - - [12/Apr/2024:21:57:54] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-12T21:57:55.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:55 smithi114 ceph-mon[25759]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:57:56.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:55 smithi097 ceph-mon[29620]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:57:56.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:55 smithi097 ceph-mon[24835]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:57:56.661 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:57:56 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: ::ffff:172.21.15.114 - - [12/Apr/2024:21:57:56] "GET /metrics HTTP/1.1" 200 214836 "" "Prometheus/2.33.4" 2024-04-12T21:57:56.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:56 smithi114 ceph-mon[25759]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T21:57:57.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:56 smithi097 ceph-mon[24835]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T21:57:57.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:56 smithi097 ceph-mon[29620]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T21:57:57.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:57 smithi114 ceph-mon[25759]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:57:58.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:57 smithi097 ceph-mon[24835]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:57:58.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:57 smithi097 ceph-mon[29620]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:57:59.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:57:59 smithi097 ceph-mon[24835]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:57:59.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:57:59 smithi097 ceph-mon[29620]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:57:59.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:57:59 smithi114 ceph-mon[25759]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:58:00.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:58:00 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 2]}]: dispatch 2024-04-12T21:58:00.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:58:00 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-04-12T21:58:00.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:58:00 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 2]}]: dispatch 2024-04-12T21:58:00.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:58:00 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-04-12T21:58:00.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:58:00 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-04-12T21:58:00.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:58:00 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-04-12T21:58:00.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:58:00 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-04-12T21:58:00.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:58:00 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-04-12T21:58:01.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:58:00 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 2]}]: dispatch 2024-04-12T21:58:01.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:58:00 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-04-12T21:58:01.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:58:00 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 2]}]: dispatch 2024-04-12T21:58:01.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:58:00 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-04-12T21:58:01.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:58:00 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-04-12T21:58:01.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:58:00 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-04-12T21:58:01.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:58:00 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-04-12T21:58:01.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:58:00 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-04-12T21:58:01.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:58:00 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 2]}]: dispatch 2024-04-12T21:58:01.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:58:00 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-04-12T21:58:01.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:58:00 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 2]}]: dispatch 2024-04-12T21:58:01.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:58:00 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-04-12T21:58:01.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:58:00 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-04-12T21:58:01.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:58:00 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-04-12T21:58:01.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:58:00 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-04-12T21:58:01.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:58:00 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-04-12T21:58:01.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:58:01 smithi114 ceph-mon[25759]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:58:01.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:58:01 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 2]}]': finished 2024-04-12T21:58:01.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:58:01 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-04-12T21:58:01.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:58:01 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-04-12T21:58:01.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:58:01 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-04-12T21:58:01.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:58:01 smithi114 ceph-mon[25759]: osdmap e95: 8 total, 8 up, 8 in 2024-04-12T21:58:01.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:58:01 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T21:58:01.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:58:01 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T21:58:01.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:58:01 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T21:58:01.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:58:01 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T21:58:01.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:58:01 smithi114 ceph-mon[25759]: osdmap e96: 8 total, 8 up, 8 in 2024-04-12T21:58:02.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:58:01 smithi097 ceph-mon[24835]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:58:02.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:58:01 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 2]}]': finished 2024-04-12T21:58:02.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:58:01 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-04-12T21:58:02.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:58:01 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-04-12T21:58:02.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:58:01 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-04-12T21:58:02.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:58:01 smithi097 ceph-mon[24835]: osdmap e95: 8 total, 8 up, 8 in 2024-04-12T21:58:02.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:58:01 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T21:58:02.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:58:01 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T21:58:02.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:58:01 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T21:58:02.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:58:01 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T21:58:02.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:58:01 smithi097 ceph-mon[24835]: osdmap e96: 8 total, 8 up, 8 in 2024-04-12T21:58:02.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:58:01 smithi097 ceph-mon[29620]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:58:02.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:58:01 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 2]}]': finished 2024-04-12T21:58:02.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:58:01 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-04-12T21:58:02.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:58:01 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-04-12T21:58:02.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:58:01 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-04-12T21:58:02.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:58:01 smithi097 ceph-mon[29620]: osdmap e95: 8 total, 8 up, 8 in 2024-04-12T21:58:02.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:58:01 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T21:58:02.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:58:01 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T21:58:02.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:58:01 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T21:58:02.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:58:01 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T21:58:02.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:58:01 smithi097 ceph-mon[29620]: osdmap e96: 8 total, 8 up, 8 in 2024-04-12T21:58:03.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:58:03 smithi114 ceph-mon[25759]: pgmap v131: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:58:04.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:58:03 smithi097 ceph-mon[24835]: pgmap v131: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:58:04.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:58:03 smithi097 ceph-mon[29620]: pgmap v131: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:58:04.744 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:58:04 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: ::ffff:172.21.15.114 - - [12/Apr/2024:21:58:04] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-12T21:58:05.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:58:05 smithi114 ceph-mon[25759]: pgmap v132: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-12T21:58:06.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:58:05 smithi097 ceph-mon[24835]: pgmap v132: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-12T21:58:06.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:58:05 smithi097 ceph-mon[29620]: pgmap v132: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-12T21:58:06.661 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:58:06 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: ::ffff:172.21.15.114 - - [12/Apr/2024:21:58:06] "GET /metrics HTTP/1.1" 200 214798 "" "Prometheus/2.33.4" 2024-04-12T21:58:06.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:58:06 smithi114 ceph-mon[25759]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T21:58:07.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:58:06 smithi097 ceph-mon[24835]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T21:58:07.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:58:06 smithi097 ceph-mon[29620]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T21:58:07.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:58:07 smithi114 ceph-mon[25759]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-04-12T21:58:08.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:58:07 smithi097 ceph-mon[24835]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-04-12T21:58:08.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:58:07 smithi097 ceph-mon[29620]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-04-12T21:58:09.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:58:09 smithi114 ceph-mon[25759]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-04-12T21:58:09.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:58:09 smithi097 ceph-mon[24835]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-04-12T21:58:09.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:58:09 smithi097 ceph-mon[29620]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-04-12T21:58:11.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:58:11 smithi114 ceph-mon[25759]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-04-12T21:58:12.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:58:11 smithi097 ceph-mon[24835]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-04-12T21:58:12.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:58:11 smithi097 ceph-mon[29620]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-04-12T21:58:13.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:58:13 smithi114 ceph-mon[25759]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 933 B/s rd, 0 B/s wr, 1 op/s; 0 B/s, 0 objects/s recovering 2024-04-12T21:58:14.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:58:13 smithi097 ceph-mon[24835]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 933 B/s rd, 0 B/s wr, 1 op/s; 0 B/s, 0 objects/s recovering 2024-04-12T21:58:14.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:58:13 smithi097 ceph-mon[29620]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 933 B/s rd, 0 B/s wr, 1 op/s; 0 B/s, 0 objects/s recovering 2024-04-12T21:58:14.744 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:58:14 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: ::ffff:172.21.15.114 - - [12/Apr/2024:21:58:14] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-12T21:58:15.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:58:15 smithi114 ceph-mon[25759]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 0 op/s; 0 B/s, 0 objects/s recovering 2024-04-12T21:58:16.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:58:15 smithi097 ceph-mon[24835]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 0 op/s; 0 B/s, 0 objects/s recovering 2024-04-12T21:58:16.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:58:15 smithi097 ceph-mon[29620]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 0 op/s; 0 B/s, 0 objects/s recovering 2024-04-12T21:58:16.661 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:58:16 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: ::ffff:172.21.15.114 - - [12/Apr/2024:21:58:16] "GET /metrics HTTP/1.1" 200 214904 "" "Prometheus/2.33.4" 2024-04-12T21:58:16.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:58:16 smithi114 ceph-mon[25759]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T21:58:17.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:58:16 smithi097 ceph-mon[24835]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T21:58:17.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:58:16 smithi097 ceph-mon[29620]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T21:58:17.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:58:17 smithi114 ceph-mon[25759]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 0 B/s, 0 objects/s recovering 2024-04-12T21:58:18.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:58:17 smithi097 ceph-mon[24835]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 0 B/s, 0 objects/s recovering 2024-04-12T21:58:18.239 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:58:17 smithi097 ceph-mon[29620]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 0 B/s, 0 objects/s recovering 2024-04-12T21:58:19.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:58:19 smithi114 ceph-mon[25759]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:58:19.902 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:58:19 smithi097 ceph-mon[24835]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:58:19.903 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:58:19 smithi097 ceph-mon[29620]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:58:20.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:58:20 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:58:20.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:58:20 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:58:20.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:58:20 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:58:20.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:58:20 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:58:20.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:58:20 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:58:20.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:58:20 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:58:20.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:58:20 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:58:20.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:58:20 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:58:20.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:58:20 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:58:21.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:58:21 smithi114 ceph-mon[25759]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:58:21.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:58:21 smithi097 ceph-mon[24835]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:58:21.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:58:21 smithi097 ceph-mon[29620]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:58:23.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:58:23 smithi114 ceph-mon[25759]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:58:24.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:58:23 smithi097 ceph-mon[24835]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:58:24.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:58:23 smithi097 ceph-mon[29620]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:58:24.744 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:58:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: ::ffff:172.21.15.114 - - [12/Apr/2024:21:58:24] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-12T21:58:25.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:58:25 smithi114 ceph-mon[25759]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:58:26.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:58:25 smithi097 ceph-mon[24835]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:58:26.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:58:25 smithi097 ceph-mon[29620]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:58:26.661 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:58:26 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: ::ffff:172.21.15.114 - - [12/Apr/2024:21:58:26] "GET /metrics HTTP/1.1" 200 214904 "" "Prometheus/2.33.4" 2024-04-12T21:58:26.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:58:26 smithi114 ceph-mon[25759]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T21:58:27.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:58:26 smithi097 ceph-mon[24835]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T21:58:27.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:58:26 smithi097 ceph-mon[29620]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T21:58:27.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:58:27 smithi114 ceph-mon[25759]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:58:28.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:58:27 smithi097 ceph-mon[24835]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:58:28.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:58:27 smithi097 ceph-mon[29620]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:58:29.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:58:29 smithi114 ceph-mon[25759]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:58:29.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:58:29 smithi097 ceph-mon[24835]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:58:29.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:58:29 smithi097 ceph-mon[29620]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:58:31.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:58:31 smithi114 ceph-mon[25759]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:58:32.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:58:31 smithi097 ceph-mon[24835]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:58:32.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:58:31 smithi097 ceph-mon[29620]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:58:33.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:58:33 smithi114 ceph-mon[25759]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:58:34.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:58:33 smithi097 ceph-mon[24835]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:58:34.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:58:33 smithi097 ceph-mon[29620]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:58:34.744 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:58:34 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: ::ffff:172.21.15.114 - - [12/Apr/2024:21:58:34] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-12T21:58:35.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:58:35 smithi114 ceph-mon[25759]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:58:36.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:58:35 smithi097 ceph-mon[24835]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:58:36.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:58:35 smithi097 ceph-mon[29620]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:58:36.661 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:58:36 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: ::ffff:172.21.15.114 - - [12/Apr/2024:21:58:36] "GET /metrics HTTP/1.1" 200 214879 "" "Prometheus/2.33.4" 2024-04-12T21:58:36.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:58:36 smithi114 ceph-mon[25759]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T21:58:37.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:58:36 smithi097 ceph-mon[24835]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T21:58:37.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:58:36 smithi097 ceph-mon[29620]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T21:58:37.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:58:37 smithi114 ceph-mon[25759]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:58:38.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:58:37 smithi097 ceph-mon[24835]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:58:38.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:58:37 smithi097 ceph-mon[29620]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:58:39.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:58:39 smithi114 ceph-mon[25759]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:58:39.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:58:39 smithi097 ceph-mon[24835]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:58:39.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:58:39 smithi097 ceph-mon[29620]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:58:41.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:58:41 smithi114 ceph-mon[25759]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:58:42.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:58:41 smithi097 ceph-mon[24835]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:58:42.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:58:41 smithi097 ceph-mon[29620]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:58:43.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:58:43 smithi114 ceph-mon[25759]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:58:44.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:58:43 smithi097 ceph-mon[24835]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:58:44.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:58:43 smithi097 ceph-mon[29620]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:58:44.744 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:58:44 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: ::ffff:172.21.15.114 - - [12/Apr/2024:21:58:44] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-12T21:58:45.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:58:45 smithi114 ceph-mon[25759]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:58:46.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:58:45 smithi097 ceph-mon[24835]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:58:46.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:58:45 smithi097 ceph-mon[29620]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:58:46.661 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:58:46 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: ::ffff:172.21.15.114 - - [12/Apr/2024:21:58:46] "GET /metrics HTTP/1.1" 200 214875 "" "Prometheus/2.33.4" 2024-04-12T21:58:46.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:58:46 smithi114 ceph-mon[25759]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T21:58:47.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:58:46 smithi097 ceph-mon[24835]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T21:58:47.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:58:46 smithi097 ceph-mon[29620]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T21:58:47.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:58:47 smithi114 ceph-mon[25759]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:58:48.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:58:47 smithi097 ceph-mon[24835]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:58:48.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:58:47 smithi097 ceph-mon[29620]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:58:49.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:58:49 smithi114 ceph-mon[25759]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:58:49.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:58:49 smithi097 ceph-mon[24835]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:58:49.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:58:49 smithi097 ceph-mon[29620]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:58:51.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:58:51 smithi114 ceph-mon[25759]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:58:52.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:58:51 smithi097 ceph-mon[24835]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:58:52.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:58:51 smithi097 ceph-mon[29620]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:58:53.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:58:53 smithi114 ceph-mon[25759]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:58:54.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:58:53 smithi097 ceph-mon[24835]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:58:54.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:58:53 smithi097 ceph-mon[29620]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:58:54.744 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:58:54 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: ::ffff:172.21.15.114 - - [12/Apr/2024:21:58:54] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-12T21:58:55.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:58:55 smithi114 ceph-mon[25759]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:58:56.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:58:55 smithi097 ceph-mon[24835]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:58:56.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:58:55 smithi097 ceph-mon[29620]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:58:56.661 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:58:56 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: ::ffff:172.21.15.114 - - [12/Apr/2024:21:58:56] "GET /metrics HTTP/1.1" 200 214875 "" "Prometheus/2.33.4" 2024-04-12T21:58:56.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:58:56 smithi114 ceph-mon[25759]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T21:58:57.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:58:56 smithi097 ceph-mon[24835]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T21:58:57.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:58:56 smithi097 ceph-mon[29620]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T21:58:57.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:58:57 smithi114 ceph-mon[25759]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:58:58.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:58:57 smithi097 ceph-mon[24835]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:58:58.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:58:57 smithi097 ceph-mon[29620]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:58:59.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:58:59 smithi114 ceph-mon[25759]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:58:59.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:58:59 smithi097 ceph-mon[24835]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:58:59.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:58:59 smithi097 ceph-mon[29620]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:59:00.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:00 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T21:59:00.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:00 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T21:59:00.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:00 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T21:59:00.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:00 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T21:59:01.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:00 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T21:59:01.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:00 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T21:59:01.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:00 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T21:59:01.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:00 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T21:59:01.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:00 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T21:59:01.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:00 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T21:59:01.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:00 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T21:59:01.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:00 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T21:59:01.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:01 smithi114 ceph-mon[25759]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:59:02.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:01 smithi097 ceph-mon[24835]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:59:02.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:01 smithi097 ceph-mon[29620]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:59:03.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:03 smithi114 ceph-mon[25759]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:59:04.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:03 smithi097 ceph-mon[24835]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:59:04.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:03 smithi097 ceph-mon[29620]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:59:04.744 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:59:04 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: ::ffff:172.21.15.114 - - [12/Apr/2024:21:59:04] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-12T21:59:06.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:05 smithi097 ceph-mon[24835]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:59:06.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:05 smithi097 ceph-mon[29620]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:59:06.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:05 smithi114 ceph-mon[25759]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:59:06.661 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:59:06 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: ::ffff:172.21.15.114 - - [12/Apr/2024:21:59:06] "GET /metrics HTTP/1.1" 200 214890 "" "Prometheus/2.33.4" 2024-04-12T21:59:07.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:06 smithi097 ceph-mon[24835]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T21:59:07.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:06 smithi097 ceph-mon[29620]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T21:59:07.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:06 smithi114 ceph-mon[25759]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T21:59:08.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:07 smithi097 ceph-mon[24835]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:59:08.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:07 smithi097 ceph-mon[29620]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:59:08.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:07 smithi114 ceph-mon[25759]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:59:08.523 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -e sha1=b277feb2c7beb4597d8ecc39d58d5c36b81f613a -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-04-12T21:59:09.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:09 smithi114 ceph-mon[25759]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:59:09.767 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:09 smithi097 ceph-mon[24835]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:59:09.767 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:09 smithi097 ceph-mon[29620]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:59:11.096 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -e sha1=b277feb2c7beb4597d8ecc39d58d5c36b81f613a -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-04-12T21:59:11.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:11 smithi097 ceph-mon[24835]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:59:11.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:11 smithi097 ceph-mon[29620]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:59:11.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:11 smithi114 ceph-mon[25759]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:59:13.615 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -e sha1=b277feb2c7beb4597d8ecc39d58d5c36b81f613a -- bash -c 'ceph config set global log_to_journald false --force' 2024-04-12T21:59:13.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:13 smithi097 ceph-mon[24835]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:59:13.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:13 smithi097 ceph-mon[29620]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:59:13.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:13 smithi114 ceph-mon[25759]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:59:14.744 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:59:14 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: ::ffff:172.21.15.114 - - [12/Apr/2024:21:59:14] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-12T21:59:15.888 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:15 smithi097 ceph-mon[24835]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:59:15.888 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:15 smithi097 ceph-mon[29620]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:59:15.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:15 smithi114 ceph-mon[25759]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:59:16.195 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -e sha1=b277feb2c7beb4597d8ecc39d58d5c36b81f613a -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-04-12T21:59:16.499 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:59:16 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: ::ffff:172.21.15.114 - - [12/Apr/2024:21:59:16] "GET /metrics HTTP/1.1" 200 214872 "" "Prometheus/2.33.4" 2024-04-12T21:59:16.751 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:16 smithi097 ceph-mon[24835]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T21:59:16.751 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:16 smithi097 ceph-mon[29620]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T21:59:16.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:16 smithi114 ceph-mon[25759]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T21:59:17.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:17 smithi114 ceph-mon[25759]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:59:18.072 INFO:teuthology.orchestra.run.smithi097.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a 2024-04-12T21:59:18.073 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:17 smithi097 ceph-mon[24835]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:59:18.073 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:17 smithi097 ceph-mon[29620]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:59:18.742 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-12T21:59:18.752 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi097.front.sepia.ceph.com 2024-04-12T21:59:18.753 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -e sha1=b277feb2c7beb4597d8ecc39d58d5c36b81f613a -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-04-12T21:59:19.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:19 smithi097 ceph-mon[24835]: from='client.24949 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:59:19.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:19 smithi097 ceph-mon[24835]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a 2024-04-12T21:59:19.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:19 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:59:19.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:19 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:59:19.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:19 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:59:19.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:19 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:59:19.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:19 smithi097 ceph-mon[24835]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a 2024-04-12T21:59:19.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:19 smithi097 ceph-mon[24835]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:59:19.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:19 smithi097 ceph-mon[29620]: from='client.24949 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:59:19.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:19 smithi097 ceph-mon[29620]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a 2024-04-12T21:59:19.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:19 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:59:19.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:19 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:59:19.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:19 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:59:19.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:19 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:59:19.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:19 smithi097 ceph-mon[29620]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a 2024-04-12T21:59:19.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:19 smithi097 ceph-mon[29620]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:59:19.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:19 smithi114 ceph-mon[25759]: from='client.24949 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:59:19.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:19 smithi114 ceph-mon[25759]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a 2024-04-12T21:59:19.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:19 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:59:19.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:19 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:59:19.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:19 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:59:19.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:19 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:59:19.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:19 smithi114 ceph-mon[25759]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a 2024-04-12T21:59:19.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:19 smithi114 ceph-mon[25759]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:59:20.982 INFO:teuthology.orchestra.run.smithi097.stdout:true 2024-04-12T21:59:21.814 INFO:teuthology.orchestra.run.smithi097.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T21:59:21.814 INFO:teuthology.orchestra.run.smithi097.stdout:alertmanager.a smithi097 *:9093,9094 running (2m) 2m ago 3m 17.7M - ba2b418f427c 756b0c8c01e1 2024-04-12T21:59:21.814 INFO:teuthology.orchestra.run.smithi097.stdout:grafana.a smithi114 *:3000 running (2m) 2m ago 2m 42.7M - 8.3.5 dad864ee21e9 daa3c7921258 2024-04-12T21:59:21.814 INFO:teuthology.orchestra.run.smithi097.stdout:iscsi.foo.smithi097.xndkxy smithi097 running (2m) 2m ago 2m 76.5M - 3.5 e1d6a67b021e 2d79ea3caa09 2024-04-12T21:59:21.814 INFO:teuthology.orchestra.run.smithi097.stdout:mgr.x smithi114 *:8443 running (7m) 2m ago 7m 411M - 17.2.0 e1d6a67b021e 670fe6214c3b 2024-04-12T21:59:21.814 INFO:teuthology.orchestra.run.smithi097.stdout:mgr.y smithi097 *:9283 running (9m) 2m ago 9m 456M - 17.2.0 e1d6a67b021e 666702929f2f 2024-04-12T21:59:21.814 INFO:teuthology.orchestra.run.smithi097.stdout:mon.a smithi097 running (9m) 2m ago 9m 46.2M 2048M 17.2.0 e1d6a67b021e 6e618120f594 2024-04-12T21:59:21.814 INFO:teuthology.orchestra.run.smithi097.stdout:mon.b smithi114 running (7m) 2m ago 7m 41.2M 2048M 17.2.0 e1d6a67b021e 47a08e70f856 2024-04-12T21:59:21.814 INFO:teuthology.orchestra.run.smithi097.stdout:mon.c smithi097 running (8m) 2m ago 8m 43.1M 2048M 17.2.0 e1d6a67b021e 679b33a29a0c 2024-04-12T21:59:21.815 INFO:teuthology.orchestra.run.smithi097.stdout:node-exporter.a smithi097 *:9100 running (3m) 2m ago 3m 15.5M - 1dbe0e931976 265c1a99d585 2024-04-12T21:59:21.815 INFO:teuthology.orchestra.run.smithi097.stdout:node-exporter.b smithi114 *:9100 running (3m) 2m ago 3m 16.1M - 1dbe0e931976 20d4da15891e 2024-04-12T21:59:21.815 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0 smithi097 running (7m) 2m ago 7m 51.0M 4016M 17.2.0 e1d6a67b021e abac1d8884fa 2024-04-12T21:59:21.815 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1 smithi097 running (6m) 2m ago 6m 50.0M 4016M 17.2.0 e1d6a67b021e 39192562d598 2024-04-12T21:59:21.815 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2 smithi097 running (6m) 2m ago 6m 48.2M 4016M 17.2.0 e1d6a67b021e 89daf8e339c3 2024-04-12T21:59:21.815 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3 smithi097 running (6m) 2m ago 6m 46.4M 4016M 17.2.0 e1d6a67b021e 4e9a0b3862d9 2024-04-12T21:59:21.815 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4 smithi114 running (5m) 2m ago 5m 47.7M 4272M 17.2.0 e1d6a67b021e fb42c0e8e575 2024-04-12T21:59:21.815 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5 smithi114 running (5m) 2m ago 5m 45.7M 4272M 17.2.0 e1d6a67b021e 3acd4a3b61e5 2024-04-12T21:59:21.815 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6 smithi114 running (5m) 2m ago 5m 45.1M 4272M 17.2.0 e1d6a67b021e 5969baf67f41 2024-04-12T21:59:21.815 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7 smithi114 running (4m) 2m ago 4m 47.2M 4272M 17.2.0 e1d6a67b021e f68340fecf2c 2024-04-12T21:59:21.815 INFO:teuthology.orchestra.run.smithi097.stdout:prometheus.a smithi114 *:9095 running (2m) 2m ago 4m 43.7M - 514e6a882f6e 7a2b6dbe5d8b 2024-04-12T21:59:21.815 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.foo.smithi097.sggtkh smithi097 *:8000 running (2m) 2m ago 2m 68.7M - 17.2.0 e1d6a67b021e a39f9b2d8509 2024-04-12T21:59:21.815 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.foo.smithi114.ulfbyo smithi114 *:8000 running (2m) 2m ago 2m 68.8M - 17.2.0 e1d6a67b021e df79a48bdd59 2024-04-12T21:59:21.816 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.smpl.smithi097.exxkgl smithi097 *:80 running (2m) 2m ago 2m 68.4M - 17.2.0 e1d6a67b021e a036c58a0099 2024-04-12T21:59:21.816 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.smpl.smithi114.lrgvph smithi114 *:80 running (2m) 2m ago 2m 70.4M - 17.2.0 e1d6a67b021e 031a5fc24237 2024-04-12T21:59:21.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:21 smithi114 ceph-mon[25759]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:59:22.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:21 smithi097 ceph-mon[24835]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:59:22.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:21 smithi097 ceph-mon[29620]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:59:22.400 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-12T21:59:22.400 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": { 2024-04-12T21:59:22.401 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-04-12T21:59:22.401 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T21:59:22.401 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": { 2024-04-12T21:59:22.401 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-12T21:59:22.401 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T21:59:22.401 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": { 2024-04-12T21:59:22.401 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-12T21:59:22.401 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T21:59:22.401 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": {}, 2024-04-12T21:59:22.401 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw": { 2024-04-12T21:59:22.401 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-12T21:59:22.401 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T21:59:22.401 INFO:teuthology.orchestra.run.smithi097.stdout: "overall": { 2024-04-12T21:59:22.401 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2024-04-12T21:59:22.402 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-12T21:59:22.402 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-12T21:59:22.822 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-12T21:59:22.822 INFO:teuthology.orchestra.run.smithi097.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a", 2024-04-12T21:59:22.823 INFO:teuthology.orchestra.run.smithi097.stdout: "in_progress": true, 2024-04-12T21:59:22.823 INFO:teuthology.orchestra.run.smithi097.stdout: "services_complete": [], 2024-04-12T21:59:22.823 INFO:teuthology.orchestra.run.smithi097.stdout: "progress": "", 2024-04-12T21:59:22.823 INFO:teuthology.orchestra.run.smithi097.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a image" 2024-04-12T21:59:22.823 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-12T21:59:22.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:22 smithi114 ceph-mon[25759]: from='client.24955 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:59:22.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:22 smithi114 ceph-mon[25759]: from='client.15066 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:59:22.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:22 smithi114 ceph-mon[25759]: from='client.15072 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:59:22.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:22 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/2284911375' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T21:59:23.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:22 smithi097 ceph-mon[24835]: from='client.24955 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:59:23.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:22 smithi097 ceph-mon[24835]: from='client.15066 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:59:23.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:22 smithi097 ceph-mon[24835]: from='client.15072 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:59:23.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:22 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/2284911375' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T21:59:23.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:22 smithi097 ceph-mon[29620]: from='client.24955 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:59:23.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:22 smithi097 ceph-mon[29620]: from='client.15066 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:59:23.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:22 smithi097 ceph-mon[29620]: from='client.15072 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:59:23.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:22 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/2284911375' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T21:59:23.288 INFO:teuthology.orchestra.run.smithi097.stdout:HEALTH_OK 2024-04-12T21:59:23.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:23 smithi114 ceph-mon[25759]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:59:23.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:23 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/2075193741' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T21:59:24.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:23 smithi097 ceph-mon[24835]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:59:24.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:23 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/2075193741' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T21:59:24.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:23 smithi097 ceph-mon[29620]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:59:24.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:23 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/2075193741' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T21:59:24.718 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:59:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: ::ffff:172.21.15.114 - - [12/Apr/2024:21:59:24] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-12T21:59:24.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:24 smithi114 ceph-mon[25759]: from='client.15084 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:59:25.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:24 smithi097 ceph-mon[24835]: from='client.15084 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:59:25.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:24 smithi097 ceph-mon[29620]: from='client.15084 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:59:25.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:25 smithi114 ceph-mon[25759]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:59:26.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:25 smithi097 ceph-mon[24835]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:59:26.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:25 smithi097 ceph-mon[29620]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:59:26.661 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:59:26 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: ::ffff:172.21.15.114 - - [12/Apr/2024:21:59:26] "GET /metrics HTTP/1.1" 200 214872 "" "Prometheus/2.33.4" 2024-04-12T21:59:26.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:26 smithi114 ceph-mon[25759]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T21:59:27.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:26 smithi097 ceph-mon[24835]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T21:59:27.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:26 smithi097 ceph-mon[29620]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T21:59:27.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:27 smithi114 ceph-mon[25759]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:59:28.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:27 smithi097 ceph-mon[24835]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:59:28.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:27 smithi097 ceph-mon[29620]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:59:29.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:29 smithi114 ceph-mon[25759]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:59:29.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:29 smithi097 ceph-mon[24835]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:59:29.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:29 smithi097 ceph-mon[29620]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:59:31.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:31 smithi114 ceph-mon[25759]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:59:32.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:31 smithi097 ceph-mon[24835]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:59:32.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:31 smithi097 ceph-mon[29620]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:59:33.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:33 smithi114 ceph-mon[25759]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:59:34.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:33 smithi097 ceph-mon[24835]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:59:34.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:33 smithi097 ceph-mon[29620]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:59:34.743 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:59:34 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: ::ffff:172.21.15.114 - - [12/Apr/2024:21:59:34] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-12T21:59:35.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:35 smithi114 ceph-mon[25759]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:59:36.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:35 smithi097 ceph-mon[24835]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:59:36.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:35 smithi097 ceph-mon[29620]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:59:36.660 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:59:36 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: ::ffff:172.21.15.114 - - [12/Apr/2024:21:59:36] "GET /metrics HTTP/1.1" 200 214850 "" "Prometheus/2.33.4" 2024-04-12T21:59:36.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:36 smithi114 ceph-mon[25759]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T21:59:37.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:36 smithi097 ceph-mon[24835]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T21:59:37.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:36 smithi097 ceph-mon[29620]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T21:59:37.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:37 smithi114 ceph-mon[25759]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:59:38.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:37 smithi097 ceph-mon[24835]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:59:38.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:37 smithi097 ceph-mon[29620]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:59:39.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:39 smithi114 ceph-mon[25759]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:59:39.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:39 smithi097 ceph-mon[24835]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:59:39.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:39 smithi097 ceph-mon[29620]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:59:41.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:41 smithi114 ceph-mon[25759]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:59:42.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:41 smithi097 ceph-mon[24835]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:59:42.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:41 smithi097 ceph-mon[29620]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:59:43.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:43 smithi097 ceph-mon[24835]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:59:43.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:43 smithi097 ceph-mon[29620]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:59:43.911 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 21:59:43 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[49677]: level=warn ts=2024-04-12T21:59:43.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.97:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.97 because it doesn't contain any IP SANs" 2024-04-12T21:59:43.911 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 21:59:43 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[49677]: level=warn ts=2024-04-12T21:59:43.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.114:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.114 because it doesn't contain any IP SANs" 2024-04-12T21:59:43.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:43 smithi114 ceph-mon[25759]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:59:44.744 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:59:44 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: ::ffff:172.21.15.114 - - [12/Apr/2024:21:59:44] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-12T21:59:45.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:45 smithi114 ceph-mon[25759]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:59:46.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:45 smithi097 ceph-mon[24835]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:59:46.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:45 smithi097 ceph-mon[29620]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:59:46.563 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:59:46 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: ::ffff:172.21.15.114 - - [12/Apr/2024:21:59:46] "GET /metrics HTTP/1.1" 200 214841 "" "Prometheus/2.33.4" 2024-04-12T21:59:46.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:46 smithi097 ceph-mon[29620]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T21:59:46.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:46 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:59:46.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:46 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:59:46.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:46 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:59:46.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:46 smithi097 ceph-mon[24835]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T21:59:46.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:46 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:59:46.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:46 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:59:46.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:46 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:59:46.936 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:46 smithi114 ceph-mon[25759]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T21:59:46.936 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:46 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:59:46.936 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:46 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:59:46.936 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:46 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T21:59:47.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:47 smithi114 ceph-mon[25759]: Upgrade: Target is version 19.0.0-2053-gb277feb2 (unknown) 2024-04-12T21:59:47.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:47 smithi114 ceph-mon[25759]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, digests ['quay.ceph.io/ceph-ci/ceph@sha256:7a231d9b74270e9fc5dba084646cc7a148bd8e84d7c66434198a233cfee0f4ec'] 2024-04-12T21:59:47.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:47 smithi114 ceph-mon[25759]: Upgrade: Need to upgrade myself (mgr.y) 2024-04-12T21:59:47.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:47 smithi114 ceph-mon[25759]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:59:48.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:47 smithi097 ceph-mon[24835]: Upgrade: Target is version 19.0.0-2053-gb277feb2 (unknown) 2024-04-12T21:59:48.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:47 smithi097 ceph-mon[24835]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, digests ['quay.ceph.io/ceph-ci/ceph@sha256:7a231d9b74270e9fc5dba084646cc7a148bd8e84d7c66434198a233cfee0f4ec'] 2024-04-12T21:59:48.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:47 smithi097 ceph-mon[24835]: Upgrade: Need to upgrade myself (mgr.y) 2024-04-12T21:59:48.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:47 smithi097 ceph-mon[24835]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:59:48.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:47 smithi097 ceph-mon[29620]: Upgrade: Target is version 19.0.0-2053-gb277feb2 (unknown) 2024-04-12T21:59:48.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:47 smithi097 ceph-mon[29620]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, digests ['quay.ceph.io/ceph-ci/ceph@sha256:7a231d9b74270e9fc5dba084646cc7a148bd8e84d7c66434198a233cfee0f4ec'] 2024-04-12T21:59:48.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:47 smithi097 ceph-mon[29620]: Upgrade: Need to upgrade myself (mgr.y) 2024-04-12T21:59:48.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:47 smithi097 ceph-mon[29620]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:59:48.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:48 smithi114 ceph-mon[25759]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a on smithi114 2024-04-12T21:59:49.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:48 smithi097 ceph-mon[29620]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a on smithi114 2024-04-12T21:59:49.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:48 smithi097 ceph-mon[24835]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a on smithi114 2024-04-12T21:59:49.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:49 smithi114 ceph-mon[25759]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:59:50.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:49 smithi097 ceph-mon[24835]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:59:50.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:49 smithi097 ceph-mon[29620]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:59:51.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:51 smithi097 ceph-mon[24835]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:59:51.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:51 smithi097 ceph-mon[29620]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:59:51.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:51 smithi114 ceph-mon[25759]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:59:53.616 INFO:teuthology.orchestra.run.smithi097.stdout:true 2024-04-12T21:59:53.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:53 smithi097 ceph-mon[24835]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:59:53.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:53 smithi097 ceph-mon[29620]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:59:53.911 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 21:59:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[49677]: level=error ts=2024-04-12T21:59:53.508Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.97:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.97 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.114:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.114 because it doesn't contain any IP SANs" 2024-04-12T21:59:53.911 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 21:59:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[49677]: level=warn ts=2024-04-12T21:59:53.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.97:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.97 because it doesn't contain any IP SANs" 2024-04-12T21:59:53.911 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 21:59:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[49677]: level=warn ts=2024-04-12T21:59:53.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.114:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.114 because it doesn't contain any IP SANs" 2024-04-12T21:59:53.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:53 smithi114 ceph-mon[25759]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:59:54.198 INFO:teuthology.orchestra.run.smithi097.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T21:59:54.199 INFO:teuthology.orchestra.run.smithi097.stdout:alertmanager.a smithi097 *:9093,9094 running (3m) 2m ago 3m 17.7M - ba2b418f427c 756b0c8c01e1 2024-04-12T21:59:54.199 INFO:teuthology.orchestra.run.smithi097.stdout:grafana.a smithi114 *:3000 running (3m) 2m ago 3m 42.7M - 8.3.5 dad864ee21e9 daa3c7921258 2024-04-12T21:59:54.199 INFO:teuthology.orchestra.run.smithi097.stdout:iscsi.foo.smithi097.xndkxy smithi097 running (2m) 2m ago 2m 76.5M - 3.5 e1d6a67b021e 2d79ea3caa09 2024-04-12T21:59:54.199 INFO:teuthology.orchestra.run.smithi097.stdout:mgr.x smithi114 *:8443 running (8m) 2m ago 8m 411M - 17.2.0 e1d6a67b021e 670fe6214c3b 2024-04-12T21:59:54.199 INFO:teuthology.orchestra.run.smithi097.stdout:mgr.y smithi097 *:9283 running (9m) 2m ago 9m 456M - 17.2.0 e1d6a67b021e 666702929f2f 2024-04-12T21:59:54.199 INFO:teuthology.orchestra.run.smithi097.stdout:mon.a smithi097 running (10m) 2m ago 10m 46.2M 2048M 17.2.0 e1d6a67b021e 6e618120f594 2024-04-12T21:59:54.199 INFO:teuthology.orchestra.run.smithi097.stdout:mon.b smithi114 running (8m) 2m ago 8m 41.2M 2048M 17.2.0 e1d6a67b021e 47a08e70f856 2024-04-12T21:59:54.199 INFO:teuthology.orchestra.run.smithi097.stdout:mon.c smithi097 running (8m) 2m ago 8m 43.1M 2048M 17.2.0 e1d6a67b021e 679b33a29a0c 2024-04-12T21:59:54.199 INFO:teuthology.orchestra.run.smithi097.stdout:node-exporter.a smithi097 *:9100 running (4m) 2m ago 4m 15.5M - 1dbe0e931976 265c1a99d585 2024-04-12T21:59:54.199 INFO:teuthology.orchestra.run.smithi097.stdout:node-exporter.b smithi114 *:9100 running (4m) 2m ago 4m 16.1M - 1dbe0e931976 20d4da15891e 2024-04-12T21:59:54.199 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0 smithi097 running (7m) 2m ago 7m 51.0M 4016M 17.2.0 e1d6a67b021e abac1d8884fa 2024-04-12T21:59:54.199 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1 smithi097 running (7m) 2m ago 7m 50.0M 4016M 17.2.0 e1d6a67b021e 39192562d598 2024-04-12T21:59:54.199 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2 smithi097 running (7m) 2m ago 7m 48.2M 4016M 17.2.0 e1d6a67b021e 89daf8e339c3 2024-04-12T21:59:54.200 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3 smithi097 running (6m) 2m ago 6m 46.4M 4016M 17.2.0 e1d6a67b021e 4e9a0b3862d9 2024-04-12T21:59:54.200 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4 smithi114 running (6m) 2m ago 6m 47.7M 4272M 17.2.0 e1d6a67b021e fb42c0e8e575 2024-04-12T21:59:54.200 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5 smithi114 running (6m) 2m ago 5m 45.7M 4272M 17.2.0 e1d6a67b021e 3acd4a3b61e5 2024-04-12T21:59:54.200 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6 smithi114 running (5m) 2m ago 5m 45.1M 4272M 17.2.0 e1d6a67b021e 5969baf67f41 2024-04-12T21:59:54.200 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7 smithi114 running (5m) 2m ago 5m 47.2M 4272M 17.2.0 e1d6a67b021e f68340fecf2c 2024-04-12T21:59:54.200 INFO:teuthology.orchestra.run.smithi097.stdout:prometheus.a smithi114 *:9095 running (3m) 2m ago 4m 43.7M - 514e6a882f6e 7a2b6dbe5d8b 2024-04-12T21:59:54.200 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.foo.smithi097.sggtkh smithi097 *:8000 running (2m) 2m ago 2m 68.7M - 17.2.0 e1d6a67b021e a39f9b2d8509 2024-04-12T21:59:54.200 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.foo.smithi114.ulfbyo smithi114 *:8000 running (2m) 2m ago 2m 68.8M - 17.2.0 e1d6a67b021e df79a48bdd59 2024-04-12T21:59:54.200 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.smpl.smithi097.exxkgl smithi097 *:80 running (2m) 2m ago 2m 68.4M - 17.2.0 e1d6a67b021e a036c58a0099 2024-04-12T21:59:54.200 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.smpl.smithi114.lrgvph smithi114 *:80 running (2m) 2m ago 2m 70.4M - 17.2.0 e1d6a67b021e 031a5fc24237 2024-04-12T21:59:54.548 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-12T21:59:54.548 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": { 2024-04-12T21:59:54.548 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-04-12T21:59:54.548 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T21:59:54.548 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": { 2024-04-12T21:59:54.548 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-12T21:59:54.548 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T21:59:54.548 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": { 2024-04-12T21:59:54.548 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-12T21:59:54.548 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T21:59:54.549 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": {}, 2024-04-12T21:59:54.549 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw": { 2024-04-12T21:59:54.549 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-12T21:59:54.549 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T21:59:54.549 INFO:teuthology.orchestra.run.smithi097.stdout: "overall": { 2024-04-12T21:59:54.549 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2024-04-12T21:59:54.549 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-12T21:59:54.549 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-12T21:59:54.728 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 21:59:54 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: ::ffff:172.21.15.114 - - [12/Apr/2024:21:59:54] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-12T21:59:54.844 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-12T21:59:54.844 INFO:teuthology.orchestra.run.smithi097.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a", 2024-04-12T21:59:54.844 INFO:teuthology.orchestra.run.smithi097.stdout: "in_progress": true, 2024-04-12T21:59:54.844 INFO:teuthology.orchestra.run.smithi097.stdout: "services_complete": [], 2024-04-12T21:59:54.844 INFO:teuthology.orchestra.run.smithi097.stdout: "progress": "0/23 daemons upgraded", 2024-04-12T21:59:54.844 INFO:teuthology.orchestra.run.smithi097.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a image on host smithi114" 2024-04-12T21:59:54.844 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-12T21:59:54.845 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:54 smithi097 ceph-mon[29620]: from='client.15096 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:59:54.845 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:54 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/98775919' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T21:59:54.845 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:54 smithi097 ceph-mon[24835]: from='client.15096 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:59:54.845 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:54 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/98775919' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T21:59:54.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:54 smithi114 ceph-mon[25759]: from='client.15096 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:59:54.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:54 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/98775919' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T21:59:55.194 INFO:teuthology.orchestra.run.smithi097.stdout:HEALTH_OK 2024-04-12T21:59:55.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:55 smithi114 ceph-mon[25759]: from='client.15102 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:59:55.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:55 smithi114 ceph-mon[25759]: from='client.25003 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:59:55.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:55 smithi114 ceph-mon[25759]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:59:55.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:55 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/1424444820' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T21:59:56.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:55 smithi097 ceph-mon[24835]: from='client.15102 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:59:56.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:55 smithi097 ceph-mon[24835]: from='client.25003 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:59:56.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:55 smithi097 ceph-mon[24835]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:59:56.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:55 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/1424444820' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T21:59:56.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:55 smithi097 ceph-mon[29620]: from='client.15102 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:59:56.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:55 smithi097 ceph-mon[29620]: from='client.25003 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:59:56.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:55 smithi097 ceph-mon[29620]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:59:56.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:55 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/1424444820' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T21:59:56.660 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 21:59:56 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: ::ffff:172.21.15.114 - - [12/Apr/2024:21:59:56] "GET /metrics HTTP/1.1" 200 214841 "" "Prometheus/2.33.4" 2024-04-12T21:59:56.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:56 smithi114 ceph-mon[25759]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:59:56.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:56 smithi114 ceph-mon[25759]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T21:59:57.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:56 smithi097 ceph-mon[24835]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:59:57.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:56 smithi097 ceph-mon[24835]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T21:59:57.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:56 smithi097 ceph-mon[29620]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:59:57.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:56 smithi097 ceph-mon[29620]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T21:59:57.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:57 smithi114 ceph-mon[25759]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:59:58.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:57 smithi097 ceph-mon[24835]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:59:58.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:57 smithi097 ceph-mon[29620]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:59:59.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 21:59:59 smithi114 ceph-mon[25759]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:59:59.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 21:59:59 smithi097 ceph-mon[24835]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T21:59:59.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 21:59:59 smithi097 ceph-mon[29620]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:00:00.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:00 smithi114 ceph-mon[25759]: overall HEALTH_OK 2024-04-12T22:00:00.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:00 smithi097 ceph-mon[24835]: overall HEALTH_OK 2024-04-12T22:00:00.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:00 smithi097 ceph-mon[29620]: overall HEALTH_OK 2024-04-12T22:00:01.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:01 smithi114 ceph-mon[25759]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:00:01.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:01 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T22:00:01.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:01 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T22:00:01.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:01 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T22:00:01.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:01 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T22:00:01.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:01 smithi097 ceph-mon[24835]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:00:01.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:01 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T22:00:01.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:01 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T22:00:01.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:01 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T22:00:01.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:01 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T22:00:01.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:01 smithi097 ceph-mon[29620]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:00:01.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:01 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T22:00:01.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:01 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T22:00:01.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:01 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T22:00:01.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:01 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T22:00:03.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:03 smithi097 ceph-mon[24835]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:00:03.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:03 smithi097 ceph-mon[29620]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:00:03.911 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:00:03 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[49677]: level=error ts=2024-04-12T22:00:03.508Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.97:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.97 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.114:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.114 because it doesn't contain any IP SANs" 2024-04-12T22:00:03.911 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:00:03 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[49677]: level=warn ts=2024-04-12T22:00:03.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.97:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.97 because it doesn't contain any IP SANs" 2024-04-12T22:00:03.911 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:00:03 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[49677]: level=warn ts=2024-04-12T22:00:03.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.114:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.114 because it doesn't contain any IP SANs" 2024-04-12T22:00:03.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:03 smithi114 ceph-mon[25759]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:00:04.743 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:04 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:00:04] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-12T22:00:05.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:05 smithi114 ceph-mon[25759]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:00:06.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:05 smithi097 ceph-mon[24835]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:00:06.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:05 smithi097 ceph-mon[29620]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:00:06.660 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:00:06 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:00:06] "GET /metrics HTTP/1.1" 200 214849 "" "Prometheus/2.33.4" 2024-04-12T22:00:06.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:06 smithi114 ceph-mon[25759]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:00:07.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:06 smithi097 ceph-mon[24835]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:00:07.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:06 smithi097 ceph-mon[29620]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:00:07.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:07 smithi114 ceph-mon[25759]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:00:08.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:07 smithi097 ceph-mon[24835]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:00:08.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:07 smithi097 ceph-mon[29620]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:00:09.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:09 smithi114 ceph-mon[25759]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:00:09.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:09 smithi097 ceph-mon[24835]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:00:09.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:09 smithi097 ceph-mon[29620]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:00:11.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:11 smithi114 ceph-mon[25759]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:00:12.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:11 smithi097 ceph-mon[24835]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:00:12.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:11 smithi097 ceph-mon[29620]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:00:13.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:13 smithi097 ceph-mon[24835]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:00:13.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:13 smithi097 ceph-mon[29620]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:00:13.911 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:00:13 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[49677]: level=error ts=2024-04-12T22:00:13.509Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.114:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.114 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.97:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.97 because it doesn't contain any IP SANs" 2024-04-12T22:00:13.912 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:00:13 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[49677]: level=warn ts=2024-04-12T22:00:13.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.97:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.97 because it doesn't contain any IP SANs" 2024-04-12T22:00:13.912 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:00:13 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[49677]: level=warn ts=2024-04-12T22:00:13.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.114:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.114 because it doesn't contain any IP SANs" 2024-04-12T22:00:13.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:13 smithi114 ceph-mon[25759]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:00:14.544 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:14 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[26939]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:00:14] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-12T22:00:15.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:15 smithi114 ceph-mon[25759]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:00:15.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:15 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T22:00:15.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:15 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T22:00:15.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:15 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T22:00:15.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:15 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T22:00:15.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:15 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:00:16.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:15 smithi097 ceph-mon[24835]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:00:16.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:15 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T22:00:16.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:15 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T22:00:16.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:15 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T22:00:16.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:15 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T22:00:16.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:15 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:00:16.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:15 smithi097 ceph-mon[29620]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:00:16.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:15 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T22:00:16.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:15 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T22:00:16.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:15 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T22:00:16.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:15 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T22:00:16.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:15 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:00:16.661 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:00:16 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:00:16] "GET /metrics HTTP/1.1" 200 214821 "" "Prometheus/2.33.4" 2024-04-12T22:00:16.993 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:16 smithi114 systemd[1]: Stopping Ceph mgr.x for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:00:16.994 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:16 smithi114 bash[46761]: Error: no container with name or ID "ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr.x" found: no such container 2024-04-12T22:00:16.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:16 smithi114 ceph-mon[25759]: Upgrade: Updating mgr.x 2024-04-12T22:00:16.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:16 smithi114 ceph-mon[25759]: Deploying daemon mgr.x on smithi114 2024-04-12T22:00:16.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:16 smithi114 ceph-mon[25759]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:00:17.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:16 smithi097 ceph-mon[24835]: Upgrade: Updating mgr.x 2024-04-12T22:00:17.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:16 smithi097 ceph-mon[24835]: Deploying daemon mgr.x on smithi114 2024-04-12T22:00:17.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:16 smithi097 ceph-mon[24835]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:00:17.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:16 smithi097 ceph-mon[29620]: Upgrade: Updating mgr.x 2024-04-12T22:00:17.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:16 smithi097 ceph-mon[29620]: Deploying daemon mgr.x on smithi114 2024-04-12T22:00:17.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:16 smithi097 ceph-mon[29620]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:00:17.483 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:17 smithi114 podman[46773]: 2024-04-12 22:00:17.095383857 +0000 UTC m=+0.125597809 container died 670fe6214c3bb60c4e535b275712d59e327e9aae2e04f18edd9c5dfbb8bb0860 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, version=8, ceph=True, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, io.openshift.expose-services=) 2024-04-12T22:00:17.735 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:17 smithi114 ceph-mon[25759]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:00:17.736 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:17 smithi114 podman[46773]: 2024-04-12 22:00:17.483028507 +0000 UTC m=+0.513242454 container cleanup 670fe6214c3bb60c4e535b275712d59e327e9aae2e04f18edd9c5dfbb8bb0860 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, vcs-type=git, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , version=8, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream) 2024-04-12T22:00:17.736 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:17 smithi114 bash[46773]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x 2024-04-12T22:00:17.736 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:17 smithi114 bash[46800]: Error: no container with name or ID "ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr.x" found: no such container 2024-04-12T22:00:17.994 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:17 smithi114 podman[46788]: 2024-04-12 22:00:17.734889005 +0000 UTC m=+0.636863124 container remove 670fe6214c3bb60c4e535b275712d59e327e9aae2e04f18edd9c5dfbb8bb0860 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x, vendor=Red Hat, Inc., vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, io.openshift.expose-services=, RELEASE=HEAD, architecture=x86_64, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, release=754, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True) 2024-04-12T22:00:17.994 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:17 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-04-12T22:00:17.994 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:17 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@mgr.x.service: Failed with result 'exit-code'. 2024-04-12T22:00:17.994 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:17 smithi114 systemd[1]: Stopped Ceph mgr.x for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:00:17.994 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:17 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@mgr.x.service: Consumed 16.506s CPU time. 2024-04-12T22:00:17.994 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:17 smithi114 systemd[1]: Starting Ceph mgr.x for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:00:18.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:17 smithi097 ceph-mon[29620]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:00:18.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:17 smithi097 ceph-mon[24835]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:00:18.423 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:18 smithi114 podman[46904]: 2024-04-12 22:00:18.061541816 +0000 UTC m=+0.018033876 image pull 668895944f0ec64b67a5b7564961d2756f2966e5d4b9338516300dbec9984951 quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a 2024-04-12T22:00:18.424 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:18 smithi114 podman[46904]: 2024-04-12 22:00:18.177379481 +0000 UTC m=+0.133871570 container create 87e5b6de56ea9c82e7a157f02a63bf1d91ee5a65e8a2ffb0476e414026bb097c (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-b277feb) 2024-04-12T22:00:18.424 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:18 smithi114 podman[46904]: 2024-04-12 22:00:18.396449618 +0000 UTC m=+0.352941661 container init 87e5b6de56ea9c82e7a157f02a63bf1d91ee5a65e8a2ffb0476e414026bb097c (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, RELEASE=squid-b277feb, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, GIT_CLEAN=True) 2024-04-12T22:00:18.424 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:18 smithi114 podman[46904]: 2024-04-12 22:00:18.399127982 +0000 UTC m=+0.355620026 container start 87e5b6de56ea9c82e7a157f02a63bf1d91ee5a65e8a2ffb0476e414026bb097c (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x, RELEASE=squid-b277feb, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-12T22:00:18.603 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:00:18 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[49677]: level=warn ts=2024-04-12T22:00:18.313Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=5 err="Post \"https://172.21.15.114:8443/api/prometheus_receiver\": dial tcp 172.21.15.114:8443: connect: connection refused" 2024-04-12T22:00:18.743 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:18 smithi114 ceph-mgr[46922]: -- 172.21.15.114:0/992988804 <== mon.2 v2:172.21.15.114:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559d6b97e5a0 con 0x559d6b952c00 2024-04-12T22:00:18.743 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:18 smithi114 bash[46904]: 87e5b6de56ea9c82e7a157f02a63bf1d91ee5a65e8a2ffb0476e414026bb097c 2024-04-12T22:00:18.744 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:18 smithi114 systemd[1]: Started Ceph mgr.x for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:00:19.243 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:18 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: 2024-04-12T22:00:18.975+0000 7f7e8e051200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-12T22:00:19.244 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:19 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: 2024-04-12T22:00:19.069+0000 7f7e8e051200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-12T22:00:19.858 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:19 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: 2024-04-12T22:00:19.756+0000 7f7e8e051200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-12T22:00:19.858 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:19 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T22:00:19.858 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:19 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:00:19.858 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:19 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:00:19.858 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:19 smithi114 ceph-mon[25759]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:00:19.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:19 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T22:00:19.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:19 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:00:19.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:19 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:00:19.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:19 smithi097 ceph-mon[24835]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:00:19.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:19 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T22:00:19.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:19 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:00:19.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:19 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:00:19.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:19 smithi097 ceph-mon[29620]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:00:20.114 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:19 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: 2024-04-12T22:00:19.855+0000 7f7e8e051200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-12T22:00:20.114 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:20 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: 2024-04-12T22:00:20.008+0000 7f7e8e051200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-12T22:00:20.114 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:20 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: 2024-04-12T22:00:20.111+0000 7f7e8e051200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-12T22:00:20.493 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:20 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: 2024-04-12T22:00:20.197+0000 7f7e8e051200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-12T22:00:20.856 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:20 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: 2024-04-12T22:00:20.548+0000 7f7e8e051200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-12T22:00:20.856 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:20 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: 2024-04-12T22:00:20.660+0000 7f7e8e051200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-12T22:00:20.856 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:20 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: 2024-04-12T22:00:20.750+0000 7f7e8e051200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-12T22:00:20.856 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:20 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: 2024-04-12T22:00:20.853+0000 7f7e8e051200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-12T22:00:21.243 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: 2024-04-12T22:00:21.034+0000 7f7e8e051200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-12T22:00:21.738 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:21 smithi114 ceph-mon[25759]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:00:22.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:21 smithi097 ceph-mon[24835]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:00:22.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:21 smithi097 ceph-mon[29620]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:00:22.308 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:22 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: 2024-04-12T22:00:22.041+0000 7f7e8e051200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-12T22:00:22.308 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:22 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: 2024-04-12T22:00:22.170+0000 7f7e8e051200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-12T22:00:22.603 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:22 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: 2024-04-12T22:00:22.511+0000 7f7e8e051200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-12T22:00:22.603 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:22 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: 2024-04-12T22:00:22.600+0000 7f7e8e051200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-12T22:00:22.993 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:22 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: 2024-04-12T22:00:22.814+0000 7f7e8e051200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-12T22:00:23.318 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:23 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: 2024-04-12T22:00:23.012+0000 7f7e8e051200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-12T22:00:23.594 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:23 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: 2024-04-12T22:00:23.315+0000 7f7e8e051200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-12T22:00:23.594 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:23 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: 2024-04-12T22:00:23.589+0000 7f7e8e051200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-12T22:00:23.868 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:23 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: 2024-04-12T22:00:23.837+0000 7f7e8e051200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-12T22:00:23.868 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:23 smithi114 ceph-mon[25759]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:00:23.868 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:23 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T22:00:23.868 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:23 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T22:00:23.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:23 smithi097 ceph-mon[24835]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:00:23.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:23 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T22:00:23.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:23 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T22:00:23.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:23 smithi097 ceph-mon[29620]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:00:23.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:23 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T22:00:23.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:23 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T22:00:23.912 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:00:23 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[49677]: level=error ts=2024-04-12T22:00:23.510Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.114:8443/api/prometheus_receiver\": dial tcp 172.21.15.114:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.97:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.97 because it doesn't contain any IP SANs" 2024-04-12T22:00:23.912 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:00:23 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[49677]: level=warn ts=2024-04-12T22:00:23.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.114:8443/api/prometheus_receiver\": dial tcp 172.21.15.114:8443: connect: connection refused" 2024-04-12T22:00:23.912 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:00:23 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[49677]: level=warn ts=2024-04-12T22:00:23.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.97:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.97 because it doesn't contain any IP SANs" 2024-04-12T22:00:24.848 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: 2024-04-12T22:00:24.664+0000 7f7e8e051200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-12T22:00:25.518 INFO:teuthology.orchestra.run.smithi097.stdout:true 2024-04-12T22:00:25.587 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:25 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: 2024-04-12T22:00:25.255+0000 7f7e8e051200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-12T22:00:25.588 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:25 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: 2024-04-12T22:00:25.360+0000 7f7e8e051200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-12T22:00:25.588 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:25 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: 2024-04-12T22:00:25.586+0000 7f7e8e051200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-12T22:00:25.803 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:25 smithi097 ceph-mon[24835]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:00:25.803 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:25 smithi097 ceph-mon[24835]: Standby manager daemon x restarted 2024-04-12T22:00:25.803 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:25 smithi097 ceph-mon[24835]: Standby manager daemon x started 2024-04-12T22:00:25.803 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:25 smithi097 ceph-mon[24835]: from='mgr.? 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-12T22:00:25.803 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:25 smithi097 ceph-mon[24835]: from='mgr.? 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-12T22:00:25.803 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:25 smithi097 ceph-mon[24835]: from='mgr.? 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-12T22:00:25.803 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:25 smithi097 ceph-mon[24835]: from='mgr.? 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-12T22:00:25.803 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:25 smithi097 ceph-mon[29620]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:00:25.804 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:25 smithi097 ceph-mon[29620]: Standby manager daemon x restarted 2024-04-12T22:00:25.804 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:25 smithi097 ceph-mon[29620]: Standby manager daemon x started 2024-04-12T22:00:25.804 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:25 smithi097 ceph-mon[29620]: from='mgr.? 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-12T22:00:25.804 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:25 smithi097 ceph-mon[29620]: from='mgr.? 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-12T22:00:25.804 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:25 smithi097 ceph-mon[29620]: from='mgr.? 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-12T22:00:25.804 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:25 smithi097 ceph-mon[29620]: from='mgr.? 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-12T22:00:25.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:25 smithi114 ceph-mon[25759]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:00:25.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:25 smithi114 ceph-mon[25759]: Standby manager daemon x restarted 2024-04-12T22:00:25.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:25 smithi114 ceph-mon[25759]: Standby manager daemon x started 2024-04-12T22:00:25.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:25 smithi114 ceph-mon[25759]: from='mgr.? 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-12T22:00:25.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:25 smithi114 ceph-mon[25759]: from='mgr.? 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-12T22:00:25.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:25 smithi114 ceph-mon[25759]: from='mgr.? 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-12T22:00:25.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:25 smithi114 ceph-mon[25759]: from='mgr.? 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-12T22:00:25.994 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:25 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: [12/Apr/2024:22:00:25] ENGINE Bus STARTING 2024-04-12T22:00:25.994 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:25 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: CherryPy Checker: 2024-04-12T22:00:25.994 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:25 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: The Application mounted at '' has an empty config. 2024-04-12T22:00:25.994 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:25 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: 2024-04-12T22:00:25.994 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:25 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: [12/Apr/2024:22:00:25] ENGINE Serving on http://:::9283 2024-04-12T22:00:25.994 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:25 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: [12/Apr/2024:22:00:25] ENGINE Bus STARTED 2024-04-12T22:00:26.109 INFO:teuthology.orchestra.run.smithi097.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T22:00:26.109 INFO:teuthology.orchestra.run.smithi097.stdout:alertmanager.a smithi097 *:9093,9094 running (3m) 3m ago 4m 17.7M - ba2b418f427c 756b0c8c01e1 2024-04-12T22:00:26.109 INFO:teuthology.orchestra.run.smithi097.stdout:grafana.a smithi114 *:3000 running (4m) 3s ago 4m 52.6M - 8.3.5 dad864ee21e9 daa3c7921258 2024-04-12T22:00:26.109 INFO:teuthology.orchestra.run.smithi097.stdout:iscsi.foo.smithi097.xndkxy smithi097 running (3m) 3m ago 3m 76.5M - 3.5 e1d6a67b021e 2d79ea3caa09 2024-04-12T22:00:26.109 INFO:teuthology.orchestra.run.smithi097.stdout:mgr.x smithi114 *:8443,9283 running (8s) 3s ago 8m 50.3M - 19.0.0-2053-gb277feb2 668895944f0e 87e5b6de56ea 2024-04-12T22:00:26.109 INFO:teuthology.orchestra.run.smithi097.stdout:mgr.y smithi097 *:9283 running (10m) 3m ago 10m 456M - 17.2.0 e1d6a67b021e 666702929f2f 2024-04-12T22:00:26.109 INFO:teuthology.orchestra.run.smithi097.stdout:mon.a smithi097 running (10m) 3m ago 10m 46.2M 2048M 17.2.0 e1d6a67b021e 6e618120f594 2024-04-12T22:00:26.109 INFO:teuthology.orchestra.run.smithi097.stdout:mon.b smithi114 running (9m) 3s ago 9m 42.3M 2048M 17.2.0 e1d6a67b021e 47a08e70f856 2024-04-12T22:00:26.109 INFO:teuthology.orchestra.run.smithi097.stdout:mon.c smithi097 running (9m) 3m ago 9m 43.1M 2048M 17.2.0 e1d6a67b021e 679b33a29a0c 2024-04-12T22:00:26.110 INFO:teuthology.orchestra.run.smithi097.stdout:node-exporter.a smithi097 *:9100 running (5m) 3m ago 5m 15.5M - 1dbe0e931976 265c1a99d585 2024-04-12T22:00:26.110 INFO:teuthology.orchestra.run.smithi097.stdout:node-exporter.b smithi114 *:9100 running (4m) 3s ago 4m 17.9M - 1dbe0e931976 20d4da15891e 2024-04-12T22:00:26.110 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0 smithi097 running (8m) 3m ago 8m 51.0M 4016M 17.2.0 e1d6a67b021e abac1d8884fa 2024-04-12T22:00:26.110 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1 smithi097 running (8m) 3m ago 8m 50.0M 4016M 17.2.0 e1d6a67b021e 39192562d598 2024-04-12T22:00:26.110 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2 smithi097 running (7m) 3m ago 7m 48.2M 4016M 17.2.0 e1d6a67b021e 89daf8e339c3 2024-04-12T22:00:26.110 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3 smithi097 running (7m) 3m ago 7m 46.4M 4016M 17.2.0 e1d6a67b021e 4e9a0b3862d9 2024-04-12T22:00:26.110 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4 smithi114 running (6m) 3s ago 6m 50.7M 4272M 17.2.0 e1d6a67b021e fb42c0e8e575 2024-04-12T22:00:26.110 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5 smithi114 running (6m) 3s ago 6m 48.9M 4272M 17.2.0 e1d6a67b021e 3acd4a3b61e5 2024-04-12T22:00:26.110 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6 smithi114 running (6m) 3s ago 6m 47.8M 4272M 17.2.0 e1d6a67b021e 5969baf67f41 2024-04-12T22:00:26.110 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7 smithi114 running (5m) 3s ago 5m 49.1M 4272M 17.2.0 e1d6a67b021e f68340fecf2c 2024-04-12T22:00:26.110 INFO:teuthology.orchestra.run.smithi097.stdout:prometheus.a smithi114 *:9095 running (3m) 3s ago 5m 66.0M - 514e6a882f6e 7a2b6dbe5d8b 2024-04-12T22:00:26.110 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.foo.smithi097.sggtkh smithi097 *:8000 running (3m) 3m ago 3m 68.7M - 17.2.0 e1d6a67b021e a39f9b2d8509 2024-04-12T22:00:26.110 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.foo.smithi114.ulfbyo smithi114 *:8000 running (3m) 3s ago 3m 69.9M - 17.2.0 e1d6a67b021e df79a48bdd59 2024-04-12T22:00:26.110 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.smpl.smithi097.exxkgl smithi097 *:80 running (3m) 3m ago 3m 68.4M - 17.2.0 e1d6a67b021e a036c58a0099 2024-04-12T22:00:26.111 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.smpl.smithi114.lrgvph smithi114 *:80 running (3m) 3s ago 3m 71.3M - 17.2.0 e1d6a67b021e 031a5fc24237 2024-04-12T22:00:26.411 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:00:26 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:00:26] "GET /metrics HTTP/1.1" 200 214821 "" "Prometheus/2.33.4" 2024-04-12T22:00:26.411 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:00:26 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[49677]: level=warn ts=2024-04-12T22:00:26.281Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=4 err="Post \"https://172.21.15.114:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.114 because it doesn't contain any IP SANs" 2024-04-12T22:00:26.460 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-12T22:00:26.460 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": { 2024-04-12T22:00:26.460 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-04-12T22:00:26.460 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:00:26.460 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": { 2024-04-12T22:00:26.460 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-04-12T22:00:26.461 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 1 2024-04-12T22:00:26.461 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:00:26.461 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": { 2024-04-12T22:00:26.461 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-12T22:00:26.461 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:00:26.461 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": {}, 2024-04-12T22:00:26.461 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw": { 2024-04-12T22:00:26.461 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-12T22:00:26.461 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:00:26.461 INFO:teuthology.orchestra.run.smithi097.stdout: "overall": { 2024-04-12T22:00:26.461 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 16, 2024-04-12T22:00:26.461 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 1 2024-04-12T22:00:26.461 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-12T22:00:26.461 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-12T22:00:26.756 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-12T22:00:26.756 INFO:teuthology.orchestra.run.smithi097.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a", 2024-04-12T22:00:26.756 INFO:teuthology.orchestra.run.smithi097.stdout: "in_progress": true, 2024-04-12T22:00:26.756 INFO:teuthology.orchestra.run.smithi097.stdout: "services_complete": [], 2024-04-12T22:00:26.757 INFO:teuthology.orchestra.run.smithi097.stdout: "progress": "1/23 daemons upgraded", 2024-04-12T22:00:26.757 INFO:teuthology.orchestra.run.smithi097.stdout: "message": "Currently upgrading mgr daemons" 2024-04-12T22:00:26.757 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-12T22:00:27.108 INFO:teuthology.orchestra.run.smithi097.stdout:HEALTH_OK 2024-04-12T22:00:27.108 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:26 smithi097 ceph-mon[24835]: from='client.25033 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:00:27.108 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:26 smithi097 ceph-mon[24835]: mgrmap e21: y(active, since 5m), standbys: x 2024-04-12T22:00:27.108 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:26 smithi097 ceph-mon[24835]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:00:27.109 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:26 smithi097 ceph-mon[24835]: from='client.15144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:00:27.109 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:26 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/1698095859' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:00:27.109 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:26 smithi097 ceph-mon[29620]: from='client.25033 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:00:27.109 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:26 smithi097 ceph-mon[29620]: mgrmap e21: y(active, since 5m), standbys: x 2024-04-12T22:00:27.109 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:26 smithi097 ceph-mon[29620]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:00:27.109 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:26 smithi097 ceph-mon[29620]: from='client.15144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:00:27.109 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:26 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/1698095859' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:00:27.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:26 smithi114 ceph-mon[25759]: from='client.25033 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:00:27.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:26 smithi114 ceph-mon[25759]: mgrmap e21: y(active, since 5m), standbys: x 2024-04-12T22:00:27.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:26 smithi114 ceph-mon[25759]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:00:27.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:26 smithi114 ceph-mon[25759]: from='client.15144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:00:27.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:26 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/1698095859' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:00:27.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:27 smithi114 ceph-mon[25759]: from='client.15150 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:00:27.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:27 smithi114 ceph-mon[25759]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:00:27.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:27 smithi114 ceph-mon[25759]: from='client.15162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:00:27.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:27 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/1603087873' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T22:00:28.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:27 smithi097 ceph-mon[24835]: from='client.15150 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:00:28.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:27 smithi097 ceph-mon[24835]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:00:28.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:27 smithi097 ceph-mon[24835]: from='client.15162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:00:28.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:27 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/1603087873' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T22:00:28.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:27 smithi097 ceph-mon[29620]: from='client.15150 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:00:28.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:27 smithi097 ceph-mon[29620]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:00:28.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:27 smithi097 ceph-mon[29620]: from='client.15162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:00:28.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:27 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/1603087873' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T22:00:29.287 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:29 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: [12/Apr/2024:22:00:29] ENGINE Bus STOPPING 2024-04-12T22:00:29.287 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:29 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: [12/Apr/2024:22:00:29] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-12T22:00:29.287 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:29 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: [12/Apr/2024:22:00:29] ENGINE Bus STOPPED 2024-04-12T22:00:29.287 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:29 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: [12/Apr/2024:22:00:29] ENGINE Bus STARTING 2024-04-12T22:00:29.287 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:29 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: [12/Apr/2024:22:00:29] ENGINE Serving on http://:::9283 2024-04-12T22:00:29.287 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:29 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: [12/Apr/2024:22:00:29] ENGINE Bus STARTED 2024-04-12T22:00:29.287 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:29 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T22:00:29.288 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:29 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T22:00:29.288 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:29 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:00:29.288 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:29 smithi114 ceph-mon[25759]: Upgrade: Need to upgrade myself (mgr.y) 2024-04-12T22:00:29.288 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:29 smithi114 ceph-mon[25759]: Failing over to other MGR 2024-04-12T22:00:29.288 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:29 smithi114 ceph-mon[25759]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-04-12T22:00:29.288 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:29 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-04-12T22:00:29.288 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:29 smithi114 ceph-mon[25759]: osdmap e97: 8 total, 8 up, 8 in 2024-04-12T22:00:29.288 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:29 smithi114 ceph-mon[25759]: Standby manager daemon y started 2024-04-12T22:00:29.288 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:29 smithi114 ceph-mon[25759]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-12T22:00:29.289 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:00:29 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: ignoring --setuser ceph since I am not root 2024-04-12T22:00:29.289 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:00:29 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: ignoring --setgroup ceph since I am not root 2024-04-12T22:00:29.289 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:00:29 smithi097 ceph-mgr[25072]: -- 172.21.15.97:0/1288655293 <== mon.1 v2:172.21.15.97:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55bcae680340 con 0x55bcae658c00 2024-04-12T22:00:29.289 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:29 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T22:00:29.289 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:29 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T22:00:29.289 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:29 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:00:29.289 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:29 smithi097 ceph-mon[24835]: Upgrade: Need to upgrade myself (mgr.y) 2024-04-12T22:00:29.289 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:29 smithi097 ceph-mon[24835]: Failing over to other MGR 2024-04-12T22:00:29.289 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:29 smithi097 ceph-mon[24835]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-04-12T22:00:29.289 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:29 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-04-12T22:00:29.290 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:29 smithi097 ceph-mon[24835]: osdmap e97: 8 total, 8 up, 8 in 2024-04-12T22:00:29.290 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:29 smithi097 ceph-mon[24835]: Standby manager daemon y started 2024-04-12T22:00:29.290 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:29 smithi097 ceph-mon[24835]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-12T22:00:29.290 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:29 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T22:00:29.290 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:29 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' 2024-04-12T22:00:29.290 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:29 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:00:29.290 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:29 smithi097 ceph-mon[29620]: Upgrade: Need to upgrade myself (mgr.y) 2024-04-12T22:00:29.290 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:29 smithi097 ceph-mon[29620]: Failing over to other MGR 2024-04-12T22:00:29.290 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:29 smithi097 ceph-mon[29620]: from='mgr.24352 172.21.15.97:0/3711913680' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-04-12T22:00:29.290 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:29 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-04-12T22:00:29.290 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:29 smithi097 ceph-mon[29620]: osdmap e97: 8 total, 8 up, 8 in 2024-04-12T22:00:29.290 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:29 smithi097 ceph-mon[29620]: Standby manager daemon y started 2024-04-12T22:00:29.291 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:29 smithi097 ceph-mon[29620]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-12T22:00:29.910 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:00:29 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T22:00:29.564+0000 7f4bcfea6000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-12T22:00:29.910 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:00:29 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T22:00:29.659+0000 7f4bcfea6000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-12T22:00:30.344 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T22:00:30.254+0000 7f4bcfea6000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-12T22:00:30.344 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[49677]: level=warn ts=2024-04-12T22:00:30.180Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=6 err="Post \"https://172.21.15.97:8443//api/prometheus_receiver\": dial tcp 172.21.15.97:8443: connect: connection refused" 2024-04-12T22:00:30.345 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[24835]: from='mgr.24352 ' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-04-12T22:00:30.345 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[24835]: mgrmap e22: x(active, starting, since 0.995584s), standbys: y 2024-04-12T22:00:30.345 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[24835]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-12T22:00:30.345 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[24835]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T22:00:30.345 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[24835]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T22:00:30.345 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[24835]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-12T22:00:30.345 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[24835]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-12T22:00:30.345 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[24835]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T22:00:30.345 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[24835]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T22:00:30.346 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[24835]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T22:00:30.346 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[24835]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T22:00:30.346 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[24835]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T22:00:30.346 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[24835]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T22:00:30.346 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[24835]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-12T22:00:30.346 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[24835]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-12T22:00:30.346 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[24835]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-12T22:00:30.346 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[24835]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-12T22:00:30.346 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[24835]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-12T22:00:30.346 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[24835]: Manager daemon x is now available 2024-04-12T22:00:30.346 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:30.346 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[24835]: Queued rgw.foo for migration 2024-04-12T22:00:30.346 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:30.347 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[24835]: Queued rgw.smpl for migration 2024-04-12T22:00:30.347 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[24835]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-04-12T22:00:30.347 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[24835]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-04-12T22:00:30.347 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:30.347 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[24835]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:00:30.347 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[24835]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-12T22:00:30.347 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[29620]: from='mgr.24352 ' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-04-12T22:00:30.347 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[29620]: mgrmap e22: x(active, starting, since 0.995584s), standbys: y 2024-04-12T22:00:30.347 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[29620]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-12T22:00:30.348 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[29620]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T22:00:30.348 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[29620]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T22:00:30.348 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[29620]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-12T22:00:30.348 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[29620]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-12T22:00:30.348 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[29620]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T22:00:30.348 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[29620]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T22:00:30.348 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[29620]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T22:00:30.348 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[29620]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T22:00:30.348 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[29620]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T22:00:30.348 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[29620]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T22:00:30.348 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[29620]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-12T22:00:30.348 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[29620]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-12T22:00:30.348 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[29620]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-12T22:00:30.348 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[29620]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-12T22:00:30.348 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[29620]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-12T22:00:30.349 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[29620]: Manager daemon x is now available 2024-04-12T22:00:30.349 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:30.349 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[29620]: Queued rgw.foo for migration 2024-04-12T22:00:30.349 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:30.349 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[29620]: Queued rgw.smpl for migration 2024-04-12T22:00:30.349 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[29620]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-04-12T22:00:30.349 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[29620]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-04-12T22:00:30.349 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:30.349 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[29620]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:00:30.349 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[29620]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-12T22:00:30.349 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-12T22:00:30.349 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[24835]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-12T22:00:30.350 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-12T22:00:30.350 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:30.350 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:30.350 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-12T22:00:30.350 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[29620]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-12T22:00:30.350 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-12T22:00:30.350 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:30.350 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:30.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:30 smithi114 ceph-mon[25759]: from='mgr.24352 ' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-04-12T22:00:30.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:30 smithi114 ceph-mon[25759]: mgrmap e22: x(active, starting, since 0.995584s), standbys: y 2024-04-12T22:00:30.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:30 smithi114 ceph-mon[25759]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-12T22:00:30.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:30 smithi114 ceph-mon[25759]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T22:00:30.495 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:30 smithi114 ceph-mon[25759]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T22:00:30.495 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:30 smithi114 ceph-mon[25759]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-12T22:00:30.495 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:30 smithi114 ceph-mon[25759]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-12T22:00:30.495 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:30 smithi114 ceph-mon[25759]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T22:00:30.495 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:30 smithi114 ceph-mon[25759]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T22:00:30.495 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:30 smithi114 ceph-mon[25759]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T22:00:30.495 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:30 smithi114 ceph-mon[25759]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T22:00:30.495 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:30 smithi114 ceph-mon[25759]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T22:00:30.495 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:30 smithi114 ceph-mon[25759]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T22:00:30.495 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:30 smithi114 ceph-mon[25759]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-12T22:00:30.495 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:30 smithi114 ceph-mon[25759]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-12T22:00:30.495 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:30 smithi114 ceph-mon[25759]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-12T22:00:30.495 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:30 smithi114 ceph-mon[25759]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-12T22:00:30.495 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:30 smithi114 ceph-mon[25759]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-12T22:00:30.495 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:30 smithi114 ceph-mon[25759]: Manager daemon x is now available 2024-04-12T22:00:30.495 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:30 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:30.496 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:30 smithi114 ceph-mon[25759]: Queued rgw.foo for migration 2024-04-12T22:00:30.496 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:30 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:30.496 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:30 smithi114 ceph-mon[25759]: Queued rgw.smpl for migration 2024-04-12T22:00:30.496 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:30 smithi114 ceph-mon[25759]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-04-12T22:00:30.496 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:30 smithi114 ceph-mon[25759]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-04-12T22:00:30.496 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:30 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:30.496 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:30 smithi114 ceph-mon[25759]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:00:30.496 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:30 smithi114 ceph-mon[25759]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-12T22:00:30.496 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:30 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-12T22:00:30.496 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:30 smithi114 ceph-mon[25759]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-12T22:00:30.496 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:30 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-12T22:00:30.496 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:30 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:30.496 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:30 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:30.656 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T22:00:30.343+0000 7f4bcfea6000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-12T22:00:30.656 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T22:00:30.427+0000 7f4bcfea6000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-12T22:00:30.656 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T22:00:30.574+0000 7f4bcfea6000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-12T22:00:30.656 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T22:00:30.655+0000 7f4bcfea6000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-12T22:00:30.910 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T22:00:30.765+0000 7f4bcfea6000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-12T22:00:30.911 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:00:30 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T22:00:30.848+0000 7f4bcfea6000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-12T22:00:31.410 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:31 smithi097 ceph-mon[29620]: Deploying cephadm binary to smithi097 2024-04-12T22:00:31.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:31 smithi097 ceph-mon[29620]: Deploying cephadm binary to smithi114 2024-04-12T22:00:31.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:31 smithi097 ceph-mon[29620]: mgrmap e23: x(active, since 2s), standbys: y 2024-04-12T22:00:31.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:31 smithi097 ceph-mon[29620]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:00:31.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:31 smithi097 ceph-mon[24835]: Deploying cephadm binary to smithi097 2024-04-12T22:00:31.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:31 smithi097 ceph-mon[24835]: Deploying cephadm binary to smithi114 2024-04-12T22:00:31.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:31 smithi097 ceph-mon[24835]: mgrmap e23: x(active, since 2s), standbys: y 2024-04-12T22:00:31.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:31 smithi097 ceph-mon[24835]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:00:31.411 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:00:31 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T22:00:31.012+0000 7f4bcfea6000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-12T22:00:31.412 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:00:31 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T22:00:31.102+0000 7f4bcfea6000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-12T22:00:31.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:31 smithi114 ceph-mon[25759]: Deploying cephadm binary to smithi097 2024-04-12T22:00:31.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:31 smithi114 ceph-mon[25759]: Deploying cephadm binary to smithi114 2024-04-12T22:00:31.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:31 smithi114 ceph-mon[25759]: mgrmap e23: x(active, since 2s), standbys: y 2024-04-12T22:00:31.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:31 smithi114 ceph-mon[25759]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:00:32.410 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:00:32 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T22:00:32.132+0000 7f4bcfea6000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-12T22:00:32.411 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:00:32 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T22:00:32.263+0000 7f4bcfea6000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-12T22:00:32.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:32 smithi097 ceph-mon[29620]: [12/Apr/2024:22:00:30] ENGINE Bus STARTING 2024-04-12T22:00:32.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:32 smithi097 ceph-mon[29620]: [12/Apr/2024:22:00:30] ENGINE Serving on https://172.21.15.114:7150 2024-04-12T22:00:32.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:32 smithi097 ceph-mon[29620]: [12/Apr/2024:22:00:30] ENGINE Serving on http://172.21.15.114:8765 2024-04-12T22:00:32.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:32 smithi097 ceph-mon[29620]: [12/Apr/2024:22:00:30] ENGINE Bus STARTED 2024-04-12T22:00:32.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:32 smithi097 ceph-mon[29620]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:00:32.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:32 smithi097 ceph-mon[24835]: [12/Apr/2024:22:00:30] ENGINE Bus STARTING 2024-04-12T22:00:32.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:32 smithi097 ceph-mon[24835]: [12/Apr/2024:22:00:30] ENGINE Serving on https://172.21.15.114:7150 2024-04-12T22:00:32.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:32 smithi097 ceph-mon[24835]: [12/Apr/2024:22:00:30] ENGINE Serving on http://172.21.15.114:8765 2024-04-12T22:00:32.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:32 smithi097 ceph-mon[24835]: [12/Apr/2024:22:00:30] ENGINE Bus STARTED 2024-04-12T22:00:32.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:32 smithi097 ceph-mon[24835]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:00:32.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:32 smithi114 ceph-mon[25759]: [12/Apr/2024:22:00:30] ENGINE Bus STARTING 2024-04-12T22:00:32.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:32 smithi114 ceph-mon[25759]: [12/Apr/2024:22:00:30] ENGINE Serving on https://172.21.15.114:7150 2024-04-12T22:00:32.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:32 smithi114 ceph-mon[25759]: [12/Apr/2024:22:00:30] ENGINE Serving on http://172.21.15.114:8765 2024-04-12T22:00:32.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:32 smithi114 ceph-mon[25759]: [12/Apr/2024:22:00:30] ENGINE Bus STARTED 2024-04-12T22:00:32.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:32 smithi114 ceph-mon[25759]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:00:32.910 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:00:32 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T22:00:32.455+0000 7f4bcfea6000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-12T22:00:32.910 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:00:32 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T22:00:32.543+0000 7f4bcfea6000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-12T22:00:32.911 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:00:32 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T22:00:32.680+0000 7f4bcfea6000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-12T22:00:33.316 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:33 smithi097 ceph-mon[24835]: mgrmap e24: x(active, since 4s), standbys: y 2024-04-12T22:00:33.316 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:33 smithi097 ceph-mon[29620]: mgrmap e24: x(active, since 4s), standbys: y 2024-04-12T22:00:33.317 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:00:33 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T22:00:33.005+0000 7f4bcfea6000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-12T22:00:33.317 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:00:33 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T22:00:33.229+0000 7f4bcfea6000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-12T22:00:33.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:33 smithi114 ceph-mon[25759]: mgrmap e24: x(active, since 4s), standbys: y 2024-04-12T22:00:33.660 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:00:33 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T22:00:33.314+0000 7f4bcfea6000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-12T22:00:33.661 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:00:33 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T22:00:33.488+0000 7f4bcfea6000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-12T22:00:33.661 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:00:33 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[49677]: level=error ts=2024-04-12T22:00:33.511Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.97:8443//api/prometheus_receiver\": dial tcp 172.21.15.97:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.114:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.114 because it doesn't contain any IP SANs" 2024-04-12T22:00:33.661 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:00:33 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[49677]: level=warn ts=2024-04-12T22:00:33.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.97:8443//api/prometheus_receiver\": dial tcp 172.21.15.97:8443: connect: connection refused" 2024-04-12T22:00:33.661 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:00:33 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[49677]: level=warn ts=2024-04-12T22:00:33.516Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.114:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.114 because it doesn't contain any IP SANs" 2024-04-12T22:00:34.373 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:34 smithi114 ceph-mon[25759]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:00:34.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:34 smithi097 ceph-mon[24835]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:00:34.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:34 smithi097 ceph-mon[29620]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:00:34.743 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:34 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:00:34] "GET /metrics HTTP/1.1" 200 35350 "" "Prometheus/2.33.4" 2024-04-12T22:00:34.753 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:00:34 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T22:00:34.426+0000 7f4bcfea6000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-12T22:00:34.753 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:00:34 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T22:00:34.751+0000 7f4bcfea6000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-12T22:00:35.008 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:00:34 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T22:00:34.851+0000 7f4bcfea6000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-12T22:00:35.008 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:00:35 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T22:00:35.006+0000 7f4bcfea6000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-12T22:00:35.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:35 smithi114 ceph-mon[25759]: Standby manager daemon y restarted 2024-04-12T22:00:35.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:35 smithi114 ceph-mon[25759]: Standby manager daemon y started 2024-04-12T22:00:35.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:35 smithi114 ceph-mon[25759]: from='mgr.? 172.21.15.97:0/664746534' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-12T22:00:35.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:35 smithi114 ceph-mon[25759]: from='mgr.? 172.21.15.97:0/664746534' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-12T22:00:35.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:35 smithi114 ceph-mon[25759]: from='mgr.? 172.21.15.97:0/664746534' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-12T22:00:35.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:35 smithi114 ceph-mon[25759]: from='mgr.? 172.21.15.97:0/664746534' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-12T22:00:35.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:35 smithi097 ceph-mon[24835]: Standby manager daemon y restarted 2024-04-12T22:00:35.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:35 smithi097 ceph-mon[24835]: Standby manager daemon y started 2024-04-12T22:00:35.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:35 smithi097 ceph-mon[24835]: from='mgr.? 172.21.15.97:0/664746534' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-12T22:00:35.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:35 smithi097 ceph-mon[24835]: from='mgr.? 172.21.15.97:0/664746534' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-12T22:00:35.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:35 smithi097 ceph-mon[24835]: from='mgr.? 172.21.15.97:0/664746534' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-12T22:00:35.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:35 smithi097 ceph-mon[24835]: from='mgr.? 172.21.15.97:0/664746534' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-12T22:00:35.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:35 smithi097 ceph-mon[29620]: Standby manager daemon y restarted 2024-04-12T22:00:35.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:35 smithi097 ceph-mon[29620]: Standby manager daemon y started 2024-04-12T22:00:35.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:35 smithi097 ceph-mon[29620]: from='mgr.? 172.21.15.97:0/664746534' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-12T22:00:35.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:35 smithi097 ceph-mon[29620]: from='mgr.? 172.21.15.97:0/664746534' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-12T22:00:35.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:35 smithi097 ceph-mon[29620]: from='mgr.? 172.21.15.97:0/664746534' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-12T22:00:35.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:35 smithi097 ceph-mon[29620]: from='mgr.? 172.21.15.97:0/664746534' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-12T22:00:35.412 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:00:35 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: [12/Apr/2024:22:00:35] ENGINE Bus STARTING 2024-04-12T22:00:35.412 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:00:35 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: CherryPy Checker: 2024-04-12T22:00:35.412 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:00:35 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: The Application mounted at '' has an empty config. 2024-04-12T22:00:35.412 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:00:35 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: 2024-04-12T22:00:35.412 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:00:35 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: [12/Apr/2024:22:00:35] ENGINE Serving on http://:::9283 2024-04-12T22:00:35.412 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:00:35 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: [12/Apr/2024:22:00:35] ENGINE Bus STARTED 2024-04-12T22:00:36.088 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:00:35 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[49677]: level=warn ts=2024-04-12T22:00:35.857Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=4 err="Post \"https://172.21.15.97:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.97 because it doesn't contain any IP SANs" 2024-04-12T22:00:36.373 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:36 smithi114 ceph-mon[25759]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:00:36.373 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:36 smithi114 ceph-mon[25759]: mgrmap e25: x(active, since 7s), standbys: y 2024-04-12T22:00:36.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:36 smithi097 ceph-mon[24835]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:00:36.445 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:36 smithi097 ceph-mon[24835]: mgrmap e25: x(active, since 7s), standbys: y 2024-04-12T22:00:36.445 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:00:36 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:00:36] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-12T22:00:36.446 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:36 smithi097 ceph-mon[29620]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:00:36.446 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:36 smithi097 ceph-mon[29620]: mgrmap e25: x(active, since 7s), standbys: y 2024-04-12T22:00:37.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:37 smithi097 ceph-mon[24835]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:00:37.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:37 smithi097 ceph-mon[29620]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:00:37.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:37 smithi114 ceph-mon[25759]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:00:38.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:38 smithi097 ceph-mon[24835]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-12T22:00:38.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:38 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:38.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:38 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:38.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:38 smithi097 ceph-mon[29620]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-12T22:00:38.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:38 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:38.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:38 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:38.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:38 smithi114 ceph-mon[25759]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-12T22:00:38.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:38 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:38.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:38 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:40.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:40 smithi097 ceph-mon[24835]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-12T22:00:40.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:40 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:40.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:40 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:40.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:40 smithi097 ceph-mon[24835]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:00:40.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:40 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:00:40.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:40 smithi097 ceph-mon[24835]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:00:40.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:40 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:00:40.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:40 smithi097 ceph-mon[24835]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:00:40.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:40 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:00:40.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:40 smithi097 ceph-mon[24835]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:00:40.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:40 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:00:40.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:40 smithi097 ceph-mon[24835]: Adjusting osd_memory_target on smithi097 to 2736M 2024-04-12T22:00:40.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:40 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:40.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:40 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:40.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:40 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:40.413 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:40 smithi097 ceph-mon[29620]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-12T22:00:40.413 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:40 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:40.413 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:40 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:40.413 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:40 smithi097 ceph-mon[29620]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:00:40.413 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:40 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:00:40.413 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:40 smithi097 ceph-mon[29620]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:00:40.413 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:40 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:00:40.413 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:40 smithi097 ceph-mon[29620]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:00:40.413 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:40 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:00:40.413 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:40 smithi097 ceph-mon[29620]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:00:40.413 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:40 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:00:40.414 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:40 smithi097 ceph-mon[29620]: Adjusting osd_memory_target on smithi097 to 2736M 2024-04-12T22:00:40.414 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:40 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:40.414 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:40 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:40.414 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:40 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:40.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:40 smithi114 ceph-mon[25759]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-12T22:00:40.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:40 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:40.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:40 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:40.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:40 smithi114 ceph-mon[25759]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:00:40.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:40 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:00:40.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:40 smithi114 ceph-mon[25759]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:00:40.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:40 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:00:40.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:40 smithi114 ceph-mon[25759]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:00:40.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:40 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:00:40.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:40 smithi114 ceph-mon[25759]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:00:40.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:40 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:00:40.495 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:40 smithi114 ceph-mon[25759]: Adjusting osd_memory_target on smithi097 to 2736M 2024-04-12T22:00:40.495 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:40 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:40.495 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:40 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:40.495 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:40 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:41.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:41 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:41.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:41 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:41.621 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:41 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:41.622 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:41 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:41.622 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:41 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:41.622 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:41 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:42.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:42 smithi114 ceph-mon[25759]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-12T22:00:42.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:42 smithi097 ceph-mon[24835]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-12T22:00:42.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:42 smithi097 ceph-mon[29620]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-12T22:00:43.910 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:00:43 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[49677]: level=error ts=2024-04-12T22:00:43.512Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.114:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.114 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.97:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.97 because it doesn't contain any IP SANs" 2024-04-12T22:00:43.911 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:00:43 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[49677]: level=warn ts=2024-04-12T22:00:43.514Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.97:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.97 because it doesn't contain any IP SANs" 2024-04-12T22:00:43.911 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:00:43 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[49677]: level=warn ts=2024-04-12T22:00:43.514Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.114:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.114 because it doesn't contain any IP SANs" 2024-04-12T22:00:44.375 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:44 smithi114 ceph-mon[25759]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-12T22:00:44.375 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:44 smithi114 ceph-mon[25759]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:00:44.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:44 smithi097 ceph-mon[24835]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-12T22:00:44.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:44 smithi097 ceph-mon[24835]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:00:44.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:44 smithi097 ceph-mon[29620]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-12T22:00:44.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:44 smithi097 ceph-mon[29620]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:00:44.743 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:44 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:00:44] "GET /metrics HTTP/1.1" 200 38154 "" "Prometheus/2.33.4" 2024-04-12T22:00:46.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:46 smithi114 ceph-mon[25759]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-12T22:00:46.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:46 smithi097 ceph-mon[24835]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-12T22:00:46.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:46 smithi097 ceph-mon[29620]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-12T22:00:46.661 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:00:46 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:00:46] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-12T22:00:47.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:47 smithi114 ceph-mon[25759]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:00:47.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:47 smithi097 ceph-mon[24835]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:00:47.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:47 smithi097 ceph-mon[29620]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:00:48.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:48 smithi114 ceph-mon[25759]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-12T22:00:48.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:48 smithi097 ceph-mon[24835]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-12T22:00:48.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:48 smithi097 ceph-mon[29620]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-12T22:00:49.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:49 smithi114 ceph-mon[25759]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:00:49.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:49 smithi097 ceph-mon[24835]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:00:49.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:49 smithi097 ceph-mon[29620]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:00:51.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:51 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:51.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:51 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:51.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:51 smithi114 ceph-mon[25759]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:00:51.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:51 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:00:51.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:51 smithi114 ceph-mon[25759]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:00:51.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:51 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:00:51.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:51 smithi114 ceph-mon[25759]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:00:51.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:51 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:00:51.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:51 smithi114 ceph-mon[25759]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:00:51.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:51 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:00:51.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:51 smithi114 ceph-mon[25759]: Adjusting osd_memory_target on smithi114 to 2992M 2024-04-12T22:00:51.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:51 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:51.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:51 smithi114 ceph-mon[25759]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:00:51.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:51 smithi114 ceph-mon[25759]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:00:51.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:51 smithi114 ceph-mon[25759]: Updating smithi097:/etc/ceph/ceph.conf 2024-04-12T22:00:51.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:51 smithi114 ceph-mon[25759]: Updating smithi114:/etc/ceph/ceph.conf 2024-04-12T22:00:51.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:51 smithi114 ceph-mon[25759]: Updating smithi097:/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/config/ceph.conf 2024-04-12T22:00:51.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:51 smithi114 ceph-mon[25759]: Updating smithi114:/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/config/ceph.conf 2024-04-12T22:00:51.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:51 smithi114 ceph-mon[25759]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:00:51.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:51 smithi114 ceph-mon[25759]: Updating smithi097:/etc/ceph/ceph.client.admin.keyring 2024-04-12T22:00:51.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:51 smithi114 ceph-mon[25759]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-04-12T22:00:52.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:51 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:52.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:51 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:52.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:51 smithi097 ceph-mon[24835]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:00:52.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:51 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:00:52.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:51 smithi097 ceph-mon[24835]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:00:52.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:51 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:00:52.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:51 smithi097 ceph-mon[24835]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:00:52.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:51 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:00:52.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:51 smithi097 ceph-mon[24835]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:00:52.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:51 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:00:52.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:51 smithi097 ceph-mon[24835]: Adjusting osd_memory_target on smithi114 to 2992M 2024-04-12T22:00:52.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:51 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:52.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:51 smithi097 ceph-mon[24835]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:00:52.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:51 smithi097 ceph-mon[24835]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:00:52.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:51 smithi097 ceph-mon[24835]: Updating smithi097:/etc/ceph/ceph.conf 2024-04-12T22:00:52.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:51 smithi097 ceph-mon[24835]: Updating smithi114:/etc/ceph/ceph.conf 2024-04-12T22:00:52.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:51 smithi097 ceph-mon[24835]: Updating smithi097:/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/config/ceph.conf 2024-04-12T22:00:52.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:51 smithi097 ceph-mon[24835]: Updating smithi114:/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/config/ceph.conf 2024-04-12T22:00:52.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:51 smithi097 ceph-mon[24835]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:00:52.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:51 smithi097 ceph-mon[24835]: Updating smithi097:/etc/ceph/ceph.client.admin.keyring 2024-04-12T22:00:52.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:51 smithi097 ceph-mon[24835]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-04-12T22:00:52.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:51 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:52.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:51 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:52.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:51 smithi097 ceph-mon[29620]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:00:52.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:51 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:00:52.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:51 smithi097 ceph-mon[29620]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:00:52.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:51 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:00:52.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:51 smithi097 ceph-mon[29620]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:00:52.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:51 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:00:52.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:51 smithi097 ceph-mon[29620]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:00:52.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:51 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:00:52.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:51 smithi097 ceph-mon[29620]: Adjusting osd_memory_target on smithi114 to 2992M 2024-04-12T22:00:52.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:51 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:52.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:51 smithi097 ceph-mon[29620]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:00:52.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:51 smithi097 ceph-mon[29620]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:00:52.164 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:51 smithi097 ceph-mon[29620]: Updating smithi097:/etc/ceph/ceph.conf 2024-04-12T22:00:52.164 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:51 smithi097 ceph-mon[29620]: Updating smithi114:/etc/ceph/ceph.conf 2024-04-12T22:00:52.164 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:51 smithi097 ceph-mon[29620]: Updating smithi097:/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/config/ceph.conf 2024-04-12T22:00:52.164 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:51 smithi097 ceph-mon[29620]: Updating smithi114:/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/config/ceph.conf 2024-04-12T22:00:52.164 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:51 smithi097 ceph-mon[29620]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:00:52.164 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:51 smithi097 ceph-mon[29620]: Updating smithi097:/etc/ceph/ceph.client.admin.keyring 2024-04-12T22:00:52.164 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:51 smithi097 ceph-mon[29620]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-04-12T22:00:52.744 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:00:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[42649]: ts=2024-04-12T22:00:52.314Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.97:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.114:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-12T22:00:52.793 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:52 smithi097 systemd[1]: Stopping Ceph node-exporter.a for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:00:52.793 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:52 smithi097 bash[58656]: Error: no container with name or ID "ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter.a" found: no such container 2024-04-12T22:00:52.793 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:52 smithi097 podman[58669]: 2024-04-12 22:00:52.57795611 +0000 UTC m=+0.113653801 container died 265c1a99d585808ff27e1ab28c03f0ddb12ac6d385ab1e487c9dcc1d9101c880 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-12T22:00:53.051 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:52 smithi097 ceph-mon[29620]: Updating smithi097:/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/config/ceph.client.admin.keyring 2024-04-12T22:00:53.051 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:52 smithi097 ceph-mon[29620]: Updating smithi114:/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/config/ceph.client.admin.keyring 2024-04-12T22:00:53.051 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:52 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:53.052 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:52 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:53.052 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:52 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:53.052 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:52 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:53.052 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:52 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:53.052 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:52 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:53.052 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:52 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:53.052 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:52 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:53.052 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:52 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:53.052 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:52 smithi097 ceph-mon[29620]: Reconfiguring node-exporter.a (dependencies changed)... 2024-04-12T22:00:53.052 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:52 smithi097 ceph-mon[29620]: Reconfiguring daemon node-exporter.a on smithi097 2024-04-12T22:00:53.052 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:52 smithi097 ceph-mon[24835]: Updating smithi097:/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/config/ceph.client.admin.keyring 2024-04-12T22:00:53.052 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:52 smithi097 ceph-mon[24835]: Updating smithi114:/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/config/ceph.client.admin.keyring 2024-04-12T22:00:53.053 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:52 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:53.053 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:52 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:53.053 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:52 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:53.053 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:52 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:53.053 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:52 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:53.053 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:52 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:53.053 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:52 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:53.053 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:52 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:53.053 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:52 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:53.053 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:52 smithi097 ceph-mon[24835]: Reconfiguring node-exporter.a (dependencies changed)... 2024-04-12T22:00:53.053 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:52 smithi097 ceph-mon[24835]: Reconfiguring daemon node-exporter.a on smithi097 2024-04-12T22:00:53.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:52 smithi114 ceph-mon[25759]: Updating smithi097:/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/config/ceph.client.admin.keyring 2024-04-12T22:00:53.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:52 smithi114 ceph-mon[25759]: Updating smithi114:/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/config/ceph.client.admin.keyring 2024-04-12T22:00:53.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:52 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:53.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:52 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:53.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:52 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:53.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:52 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:53.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:52 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:53.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:52 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:53.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:52 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:53.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:52 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:53.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:52 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:53.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:52 smithi114 ceph-mon[25759]: Reconfiguring node-exporter.a (dependencies changed)... 2024-04-12T22:00:53.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:52 smithi114 ceph-mon[25759]: Reconfiguring daemon node-exporter.a on smithi097 2024-04-12T22:00:53.410 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 podman[58669]: 2024-04-12 22:00:53.051053391 +0000 UTC m=+0.586751072 container cleanup 265c1a99d585808ff27e1ab28c03f0ddb12ac6d385ab1e487c9dcc1d9101c880 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-12T22:00:53.411 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 bash[58669]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a 2024-04-12T22:00:53.411 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 bash[58696]: Error: no container with name or ID "ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter.a" found: no such container 2024-04-12T22:00:53.662 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:00:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[49677]: level=error ts=2024-04-12T22:00:53.513Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.114:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.114 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.97:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.97 because it doesn't contain any IP SANs" 2024-04-12T22:00:53.663 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:00:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[49677]: level=warn ts=2024-04-12T22:00:53.515Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.97:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.97 because it doesn't contain any IP SANs" 2024-04-12T22:00:53.663 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:00:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[49677]: level=warn ts=2024-04-12T22:00:53.515Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.114:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.114 because it doesn't contain any IP SANs" 2024-04-12T22:00:53.663 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 podman[58683]: 2024-04-12 22:00:53.411189437 +0000 UTC m=+0.836759968 container remove 265c1a99d585808ff27e1ab28c03f0ddb12ac6d385ab1e487c9dcc1d9101c880 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-12T22:00:53.663 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-04-12T22:00:53.663 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-04-12T22:00:53.663 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 systemd[1]: Stopped Ceph node-exporter.a for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:00:53.663 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@node-exporter.a.service: Consumed 2.637s CPU time. 2024-04-12T22:00:53.663 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 systemd[1]: Starting Ceph node-exporter.a for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:00:53.663 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 podman[58767]: 2024-04-12 22:00:53.536186095 +0000 UTC m=+0.017132343 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-04-12T22:00:53.663 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 podman[58767]: 2024-04-12 22:00:53.662334371 +0000 UTC m=+0.143280624 container create 5edff9053b37195db0d2bbb5a5068e9d434a8737f6d3a4477c0b07df545e2423 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-12T22:00:53.924 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:53 smithi097 ceph-mon[24835]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:00:53.924 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:53 smithi097 ceph-mon[29620]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:00:54.200 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 podman[58767]: 2024-04-12 22:00:53.923654587 +0000 UTC m=+0.404600833 container init 5edff9053b37195db0d2bbb5a5068e9d434a8737f6d3a4477c0b07df545e2423 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-12T22:00:54.200 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 podman[58767]: 2024-04-12 22:00:53.926475742 +0000 UTC m=+0.407421998 container start 5edff9053b37195db0d2bbb5a5068e9d434a8737f6d3a4477c0b07df545e2423 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-12T22:00:54.200 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[58782]: ts=2024-04-12T22:00:53.929Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-04-12T22:00:54.200 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[58782]: ts=2024-04-12T22:00:53.930Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-04-12T22:00:54.201 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[58782]: ts=2024-04-12T22:00:53.930Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-04-12T22:00:54.201 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[58782]: ts=2024-04-12T22:00:53.930Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-04-12T22:00:54.201 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[58782]: ts=2024-04-12T22:00:53.931Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-04-12T22:00:54.201 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[58782]: ts=2024-04-12T22:00:53.931Z caller=node_exporter.go:115 level=info collector=arp 2024-04-12T22:00:54.201 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[58782]: ts=2024-04-12T22:00:53.931Z caller=node_exporter.go:115 level=info collector=bcache 2024-04-12T22:00:54.201 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[58782]: ts=2024-04-12T22:00:53.931Z caller=node_exporter.go:115 level=info collector=bonding 2024-04-12T22:00:54.201 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[58782]: ts=2024-04-12T22:00:53.931Z caller=node_exporter.go:115 level=info collector=btrfs 2024-04-12T22:00:54.202 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[58782]: ts=2024-04-12T22:00:53.931Z caller=node_exporter.go:115 level=info collector=conntrack 2024-04-12T22:00:54.202 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[58782]: ts=2024-04-12T22:00:53.931Z caller=node_exporter.go:115 level=info collector=cpu 2024-04-12T22:00:54.202 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[58782]: ts=2024-04-12T22:00:53.931Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-04-12T22:00:54.202 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[58782]: ts=2024-04-12T22:00:53.931Z caller=node_exporter.go:115 level=info collector=diskstats 2024-04-12T22:00:54.202 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[58782]: ts=2024-04-12T22:00:53.931Z caller=node_exporter.go:115 level=info collector=dmi 2024-04-12T22:00:54.202 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[58782]: ts=2024-04-12T22:00:53.931Z caller=node_exporter.go:115 level=info collector=edac 2024-04-12T22:00:54.203 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[58782]: ts=2024-04-12T22:00:53.931Z caller=node_exporter.go:115 level=info collector=entropy 2024-04-12T22:00:54.203 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[58782]: ts=2024-04-12T22:00:53.931Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-04-12T22:00:54.203 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[58782]: ts=2024-04-12T22:00:53.931Z caller=node_exporter.go:115 level=info collector=filefd 2024-04-12T22:00:54.203 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[58782]: ts=2024-04-12T22:00:53.931Z caller=node_exporter.go:115 level=info collector=filesystem 2024-04-12T22:00:54.203 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[58782]: ts=2024-04-12T22:00:53.931Z caller=node_exporter.go:115 level=info collector=hwmon 2024-04-12T22:00:54.204 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[58782]: ts=2024-04-12T22:00:53.931Z caller=node_exporter.go:115 level=info collector=infiniband 2024-04-12T22:00:54.204 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[58782]: ts=2024-04-12T22:00:53.931Z caller=node_exporter.go:115 level=info collector=ipvs 2024-04-12T22:00:54.204 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[58782]: ts=2024-04-12T22:00:53.931Z caller=node_exporter.go:115 level=info collector=loadavg 2024-04-12T22:00:54.204 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[58782]: ts=2024-04-12T22:00:53.931Z caller=node_exporter.go:115 level=info collector=mdadm 2024-04-12T22:00:54.204 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[58782]: ts=2024-04-12T22:00:53.931Z caller=node_exporter.go:115 level=info collector=meminfo 2024-04-12T22:00:54.204 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[58782]: ts=2024-04-12T22:00:53.931Z caller=node_exporter.go:115 level=info collector=netclass 2024-04-12T22:00:54.204 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[58782]: ts=2024-04-12T22:00:53.931Z caller=node_exporter.go:115 level=info collector=netdev 2024-04-12T22:00:54.204 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[58782]: ts=2024-04-12T22:00:53.931Z caller=node_exporter.go:115 level=info collector=netstat 2024-04-12T22:00:54.204 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[58782]: ts=2024-04-12T22:00:53.931Z caller=node_exporter.go:115 level=info collector=nfs 2024-04-12T22:00:54.205 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[58782]: ts=2024-04-12T22:00:53.931Z caller=node_exporter.go:115 level=info collector=nfsd 2024-04-12T22:00:54.205 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[58782]: ts=2024-04-12T22:00:53.931Z caller=node_exporter.go:115 level=info collector=nvme 2024-04-12T22:00:54.205 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[58782]: ts=2024-04-12T22:00:53.931Z caller=node_exporter.go:115 level=info collector=os 2024-04-12T22:00:54.205 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[58782]: ts=2024-04-12T22:00:53.931Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-04-12T22:00:54.205 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[58782]: ts=2024-04-12T22:00:53.931Z caller=node_exporter.go:115 level=info collector=pressure 2024-04-12T22:00:54.205 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[58782]: ts=2024-04-12T22:00:53.931Z caller=node_exporter.go:115 level=info collector=rapl 2024-04-12T22:00:54.205 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[58782]: ts=2024-04-12T22:00:53.931Z caller=node_exporter.go:115 level=info collector=schedstat 2024-04-12T22:00:54.205 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[58782]: ts=2024-04-12T22:00:53.931Z caller=node_exporter.go:115 level=info collector=sockstat 2024-04-12T22:00:54.205 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[58782]: ts=2024-04-12T22:00:53.931Z caller=node_exporter.go:115 level=info collector=softnet 2024-04-12T22:00:54.205 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[58782]: ts=2024-04-12T22:00:53.931Z caller=node_exporter.go:115 level=info collector=stat 2024-04-12T22:00:54.205 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[58782]: ts=2024-04-12T22:00:53.931Z caller=node_exporter.go:115 level=info collector=tapestats 2024-04-12T22:00:54.205 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[58782]: ts=2024-04-12T22:00:53.931Z caller=node_exporter.go:115 level=info collector=textfile 2024-04-12T22:00:54.205 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[58782]: ts=2024-04-12T22:00:53.931Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-04-12T22:00:54.205 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[58782]: ts=2024-04-12T22:00:53.931Z caller=node_exporter.go:115 level=info collector=time 2024-04-12T22:00:54.205 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[58782]: ts=2024-04-12T22:00:53.931Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-04-12T22:00:54.206 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[58782]: ts=2024-04-12T22:00:53.931Z caller=node_exporter.go:115 level=info collector=uname 2024-04-12T22:00:54.206 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[58782]: ts=2024-04-12T22:00:53.931Z caller=node_exporter.go:115 level=info collector=vmstat 2024-04-12T22:00:54.206 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[58782]: ts=2024-04-12T22:00:53.931Z caller=node_exporter.go:115 level=info collector=xfs 2024-04-12T22:00:54.206 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[58782]: ts=2024-04-12T22:00:53.931Z caller=node_exporter.go:115 level=info collector=zfs 2024-04-12T22:00:54.206 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[58782]: ts=2024-04-12T22:00:53.931Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-04-12T22:00:54.206 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a[58782]: ts=2024-04-12T22:00:53.931Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-04-12T22:00:54.206 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:54 smithi097 bash[58767]: 5edff9053b37195db0d2bbb5a5068e9d434a8737f6d3a4477c0b07df545e2423 2024-04-12T22:00:54.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:53 smithi114 ceph-mon[25759]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:00:54.660 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:00:54 smithi097 systemd[1]: Started Ceph node-exporter.a for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:00:54.743 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:00:54 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:00:54] "GET /metrics HTTP/1.1" 200 38154 "" "Prometheus/2.33.4" 2024-04-12T22:00:55.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:55 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:55.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:55 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:55.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:55 smithi114 ceph-mon[25759]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-12T22:00:55.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:55 smithi114 ceph-mon[25759]: Reconfiguring daemon alertmanager.a on smithi097 2024-04-12T22:00:55.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:55 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:55.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:55 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:55.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:55 smithi097 ceph-mon[29620]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-12T22:00:55.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:55 smithi097 ceph-mon[29620]: Reconfiguring daemon alertmanager.a on smithi097 2024-04-12T22:00:55.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:55 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:55.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:55 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:00:55.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:55 smithi097 ceph-mon[24835]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-12T22:00:55.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:55 smithi097 ceph-mon[24835]: Reconfiguring daemon alertmanager.a on smithi097 2024-04-12T22:00:56.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:56 smithi097 ceph-mon[24835]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:00:56.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:56 smithi097 ceph-mon[29620]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:00:56.661 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:00:56 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:00:56] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-12T22:00:56.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:56 smithi114 ceph-mon[25759]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:00:57.467 INFO:teuthology.orchestra.run.smithi097.stdout:true 2024-04-12T22:00:57.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:57 smithi097 ceph-mon[24835]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:00:57.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:57 smithi097 ceph-mon[29620]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:00:57.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:57 smithi114 ceph-mon[25759]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:00:58.137 INFO:teuthology.orchestra.run.smithi097.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T22:00:58.137 INFO:teuthology.orchestra.run.smithi097.stdout:alertmanager.a smithi097 *:9093,9094 running (4m) 20s ago 4m 25.8M - 0.23.0 ba2b418f427c 756b0c8c01e1 2024-04-12T22:00:58.137 INFO:teuthology.orchestra.run.smithi097.stdout:grafana.a smithi114 *:3000 running (4m) 18s ago 4m 52.7M - 8.3.5 dad864ee21e9 daa3c7921258 2024-04-12T22:00:58.137 INFO:teuthology.orchestra.run.smithi097.stdout:iscsi.foo.smithi097.xndkxy smithi097 running (3m) 20s ago 3m 53.6M - 3.5 e1d6a67b021e 2d79ea3caa09 2024-04-12T22:00:58.137 INFO:teuthology.orchestra.run.smithi097.stdout:mgr.x smithi114 *:8443,9283 running (40s) 18s ago 9m 494M - 19.0.0-2053-gb277feb2 668895944f0e 87e5b6de56ea 2024-04-12T22:00:58.138 INFO:teuthology.orchestra.run.smithi097.stdout:mgr.y smithi097 *:9283 running (11m) 20s ago 11m 382M - 17.2.0 e1d6a67b021e 666702929f2f 2024-04-12T22:00:58.138 INFO:teuthology.orchestra.run.smithi097.stdout:mon.a smithi097 running (11m) 20s ago 11m 53.8M 2048M 17.2.0 e1d6a67b021e 6e618120f594 2024-04-12T22:00:58.138 INFO:teuthology.orchestra.run.smithi097.stdout:mon.b smithi114 running (9m) 18s ago 9m 46.5M 2048M 17.2.0 e1d6a67b021e 47a08e70f856 2024-04-12T22:00:58.138 INFO:teuthology.orchestra.run.smithi097.stdout:mon.c smithi097 running (9m) 20s ago 9m 47.0M 2048M 17.2.0 e1d6a67b021e 679b33a29a0c 2024-04-12T22:00:58.138 INFO:teuthology.orchestra.run.smithi097.stdout:node-exporter.a smithi097 *:9100 starting - - - - 2024-04-12T22:00:58.138 INFO:teuthology.orchestra.run.smithi097.stdout:node-exporter.b smithi114 *:9100 running (5m) 18s ago 5m 17.8M - 1.3.1 1dbe0e931976 20d4da15891e 2024-04-12T22:00:58.138 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0 smithi097 running (8m) 20s ago 8m 53.1M 2736M 17.2.0 e1d6a67b021e abac1d8884fa 2024-04-12T22:00:58.138 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1 smithi097 running (8m) 20s ago 8m 52.0M 2736M 17.2.0 e1d6a67b021e 39192562d598 2024-04-12T22:00:58.138 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2 smithi097 running (8m) 20s ago 8m 49.9M 2736M 17.2.0 e1d6a67b021e 89daf8e339c3 2024-04-12T22:00:58.138 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3 smithi097 running (7m) 20s ago 7m 50.2M 2736M 17.2.0 e1d6a67b021e 4e9a0b3862d9 2024-04-12T22:00:58.138 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4 smithi114 running (7m) 18s ago 7m 51.4M 2992M 17.2.0 e1d6a67b021e fb42c0e8e575 2024-04-12T22:00:58.138 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5 smithi114 running (7m) 18s ago 7m 49.3M 2992M 17.2.0 e1d6a67b021e 3acd4a3b61e5 2024-04-12T22:00:58.138 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6 smithi114 running (6m) 18s ago 6m 48.4M 2992M 17.2.0 e1d6a67b021e 5969baf67f41 2024-04-12T22:00:58.139 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7 smithi114 running (6m) 18s ago 6m 49.6M 2992M 17.2.0 e1d6a67b021e f68340fecf2c 2024-04-12T22:00:58.139 INFO:teuthology.orchestra.run.smithi097.stdout:prometheus.a smithi114 *:9095 running (4m) 18s ago 5m 66.2M - 2.33.4 514e6a882f6e 7a2b6dbe5d8b 2024-04-12T22:00:58.139 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.foo.smithi097.sggtkh smithi097 *:8000 running (4m) 20s ago 4m 70.4M - 17.2.0 e1d6a67b021e a39f9b2d8509 2024-04-12T22:00:58.139 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.foo.smithi114.ulfbyo smithi114 *:8000 running (4m) 18s ago 3m 70.9M - 17.2.0 e1d6a67b021e df79a48bdd59 2024-04-12T22:00:58.139 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.smpl.smithi097.exxkgl smithi097 *:80 running (3m) 20s ago 3m 70.5M - 17.2.0 e1d6a67b021e a036c58a0099 2024-04-12T22:00:58.139 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.smpl.smithi114.lrgvph smithi114 *:80 running (3m) 18s ago 3m 72.4M - 17.2.0 e1d6a67b021e 031a5fc24237 2024-04-12T22:00:58.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:58 smithi097 ceph-mon[24835]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:00:58.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:58 smithi097 ceph-mon[29620]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:00:58.563 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-12T22:00:58.564 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": { 2024-04-12T22:00:58.564 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-04-12T22:00:58.564 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:00:58.564 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": { 2024-04-12T22:00:58.564 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-04-12T22:00:58.564 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 1 2024-04-12T22:00:58.564 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:00:58.564 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": { 2024-04-12T22:00:58.564 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-12T22:00:58.564 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:00:58.564 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": {}, 2024-04-12T22:00:58.564 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw": { 2024-04-12T22:00:58.564 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-12T22:00:58.565 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:00:58.565 INFO:teuthology.orchestra.run.smithi097.stdout: "overall": { 2024-04-12T22:00:58.565 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 16, 2024-04-12T22:00:58.565 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 1 2024-04-12T22:00:58.565 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-12T22:00:58.565 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-12T22:00:58.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:58 smithi114 ceph-mon[25759]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:00:58.903 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-12T22:00:58.903 INFO:teuthology.orchestra.run.smithi097.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a", 2024-04-12T22:00:58.903 INFO:teuthology.orchestra.run.smithi097.stdout: "in_progress": true, 2024-04-12T22:00:58.903 INFO:teuthology.orchestra.run.smithi097.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T22:00:58.903 INFO:teuthology.orchestra.run.smithi097.stdout: "services_complete": [], 2024-04-12T22:00:58.903 INFO:teuthology.orchestra.run.smithi097.stdout: "progress": "1/23 daemons upgraded", 2024-04-12T22:00:58.903 INFO:teuthology.orchestra.run.smithi097.stdout: "message": "", 2024-04-12T22:00:58.903 INFO:teuthology.orchestra.run.smithi097.stdout: "is_paused": false 2024-04-12T22:00:58.904 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-12T22:00:59.312 INFO:teuthology.orchestra.run.smithi097.stdout:HEALTH_OK 2024-04-12T22:00:59.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:59 smithi097 ceph-mon[24835]: from='client.25099 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:00:59.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:59 smithi097 ceph-mon[24835]: from='client.15210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:00:59.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:59 smithi097 ceph-mon[24835]: from='client.15216 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:00:59.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:59 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/3190385431' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:00:59.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:00:59 smithi097 ceph-mon[24835]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:00:59.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:59 smithi097 ceph-mon[29620]: from='client.25099 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:00:59.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:59 smithi097 ceph-mon[29620]: from='client.15210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:00:59.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:59 smithi097 ceph-mon[29620]: from='client.15216 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:00:59.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:59 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/3190385431' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:00:59.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:00:59 smithi097 ceph-mon[29620]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:00:59.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:59 smithi114 ceph-mon[25759]: from='client.25099 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:00:59.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:59 smithi114 ceph-mon[25759]: from='client.15210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:00:59.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:59 smithi114 ceph-mon[25759]: from='client.15216 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:00:59.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:59 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/3190385431' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:00:59.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:00:59 smithi114 ceph-mon[25759]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:01:00.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:00 smithi097 ceph-mon[24835]: from='client.15228 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:01:00.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:00 smithi097 ceph-mon[24835]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:01:00.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:00 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/1452340826' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T22:01:00.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:00 smithi097 ceph-mon[29620]: from='client.15228 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:01:00.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:00 smithi097 ceph-mon[29620]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:01:00.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:00 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/1452340826' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T22:01:00.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:00 smithi114 ceph-mon[25759]: from='client.15228 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:01:00.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:00 smithi114 ceph-mon[25759]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:01:00.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:00 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/1452340826' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T22:01:02.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:02 smithi097 ceph-mon[24835]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:01:02.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:02 smithi097 ceph-mon[29620]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:01:02.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:02 smithi114 ceph-mon[25759]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:01:02.743 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:01:02 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[42649]: ts=2024-04-12T22:01:02.314Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.97:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.114:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-12T22:01:03.117 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:01:02 smithi097 systemd[1]: Stopping Ceph alertmanager.a for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:01:03.117 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:01:02 smithi097 bash[59193]: Error: no container with name or ID "ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager.a" found: no such container 2024-04-12T22:01:03.409 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:01:03 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[49677]: level=info ts=2024-04-12T22:01:03.116Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-04-12T22:01:03.409 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:01:03 smithi097 podman[59204]: 2024-04-12 22:01:03.146720759 +0000 UTC m=+0.151191939 container died 756b0c8c01e197e3da2e10cd7e1e6b572bc1c02bb51c96a15421505db564bc25 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-12T22:01:03.659 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:01:03 smithi097 podman[59204]: 2024-04-12 22:01:03.617032849 +0000 UTC m=+0.621504072 container cleanup 756b0c8c01e197e3da2e10cd7e1e6b572bc1c02bb51c96a15421505db564bc25 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-12T22:01:03.659 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:01:03 smithi097 bash[59204]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a 2024-04-12T22:01:03.659 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:01:03 smithi097 bash[59232]: Error: no container with name or ID "ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager.a" found: no such container 2024-04-12T22:01:04.160 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:01:03 smithi097 podman[59219]: 2024-04-12 22:01:03.859657449 +0000 UTC m=+0.718002249 container remove 756b0c8c01e197e3da2e10cd7e1e6b572bc1c02bb51c96a15421505db564bc25 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-12T22:01:04.161 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:01:03 smithi097 podman[59219]: 2024-04-12 22:01:03.943595385 +0000 UTC m=+0.801940184 volume remove 6c7e166be94e9c6a41dedbab10931a486570c7d56c3abd315737938621ab0be5 2024-04-12T22:01:04.161 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:01:03 smithi097 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-04-12T22:01:04.161 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:01:03 smithi097 systemd[1]: Stopped Ceph alertmanager.a for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:01:04.161 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:01:03 smithi097 systemd[1]: Starting Ceph alertmanager.a for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:01:04.600 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:04 smithi097 ceph-mon[24835]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:01:04.601 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:04 smithi097 ceph-mon[29620]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:01:04.601 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:01:04 smithi097 podman[59300]: 2024-04-12 22:01:04.084282985 +0000 UTC m=+0.019140895 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-04-12T22:01:04.601 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:01:04 smithi097 podman[59300]: 2024-04-12 22:01:04.182390266 +0000 UTC m=+0.117248176 volume create d6ee338d9e02c888fd6788570a1a335ebc03fcae2e1269ae1d1f3bc0aa266c05 2024-04-12T22:01:04.601 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:01:04 smithi097 podman[59300]: 2024-04-12 22:01:04.403408015 +0000 UTC m=+0.338265964 container create 7b4c491fe0a09dfaa984d0ebaf9e8372d860e9bd97be6aec60fec459caf95983 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-12T22:01:04.743 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:04 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:01:04] "GET /metrics HTTP/1.1" 200 38147 "" "Prometheus/2.33.4" 2024-04-12T22:01:04.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:04 smithi114 ceph-mon[25759]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:01:04.911 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:01:04 smithi097 podman[59300]: 2024-04-12 22:01:04.697824463 +0000 UTC m=+0.632682383 container init 7b4c491fe0a09dfaa984d0ebaf9e8372d860e9bd97be6aec60fec459caf95983 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-12T22:01:04.911 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:01:04 smithi097 podman[59300]: 2024-04-12 22:01:04.701440559 +0000 UTC m=+0.636298472 container start 7b4c491fe0a09dfaa984d0ebaf9e8372d860e9bd97be6aec60fec459caf95983 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-12T22:01:04.911 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:01:04 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[59315]: level=info ts=2024-04-12T22:01:04.711Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-04-12T22:01:04.911 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:01:04 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[59315]: level=info ts=2024-04-12T22:01:04.711Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-04-12T22:01:04.911 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:01:04 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[59315]: level=info ts=2024-04-12T22:01:04.712Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.97 port=9094 2024-04-12T22:01:04.911 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:01:04 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[59315]: level=info ts=2024-04-12T22:01:04.714Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-04-12T22:01:04.911 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:01:04 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[59315]: level=info ts=2024-04-12T22:01:04.738Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-12T22:01:04.911 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:01:04 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[59315]: level=info ts=2024-04-12T22:01:04.739Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-12T22:01:04.911 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:01:04 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[59315]: level=info ts=2024-04-12T22:01:04.741Z caller=main.go:518 msg=Listening address=:9093 2024-04-12T22:01:04.911 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:01:04 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[59315]: level=info ts=2024-04-12T22:01:04.741Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-04-12T22:01:04.911 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:01:04 smithi097 bash[59300]: 7b4c491fe0a09dfaa984d0ebaf9e8372d860e9bd97be6aec60fec459caf95983 2024-04-12T22:01:04.911 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:01:04 smithi097 systemd[1]: Started Ceph alertmanager.a for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:01:05.995 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:05 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:05.995 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:05 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:05.996 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:05 smithi097 ceph-mon[29620]: Reconfiguring iscsi.foo.smithi097.xndkxy (dependencies changed)... 2024-04-12T22:01:05.996 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:05 smithi097 ceph-mon[29620]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi097.xndkxy", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-12T22:01:05.996 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:05 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi097.xndkxy", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-12T22:01:05.996 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:05 smithi097 ceph-mon[29620]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:01:05.996 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:05 smithi097 ceph-mon[29620]: Reconfiguring daemon iscsi.foo.smithi097.xndkxy on smithi097 2024-04-12T22:01:05.996 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:05 smithi097 ceph-mon[29620]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:01:05.996 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:05 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:05.996 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:05 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:05.996 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:05 smithi097 ceph-mon[24835]: Reconfiguring iscsi.foo.smithi097.xndkxy (dependencies changed)... 2024-04-12T22:01:05.996 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:05 smithi097 ceph-mon[24835]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi097.xndkxy", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-12T22:01:05.997 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:05 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi097.xndkxy", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-12T22:01:05.997 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:05 smithi097 ceph-mon[24835]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:01:05.997 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:05 smithi097 ceph-mon[24835]: Reconfiguring daemon iscsi.foo.smithi097.xndkxy on smithi097 2024-04-12T22:01:05.997 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:05 smithi097 ceph-mon[24835]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:01:06.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:05 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:06.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:05 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:06.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:05 smithi114 ceph-mon[25759]: Reconfiguring iscsi.foo.smithi097.xndkxy (dependencies changed)... 2024-04-12T22:01:06.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:05 smithi114 ceph-mon[25759]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi097.xndkxy", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-12T22:01:06.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:05 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi097.xndkxy", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-12T22:01:06.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:05 smithi114 ceph-mon[25759]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:01:06.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:05 smithi114 ceph-mon[25759]: Reconfiguring daemon iscsi.foo.smithi097.xndkxy on smithi097 2024-04-12T22:01:06.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:05 smithi114 ceph-mon[25759]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:01:06.410 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:01:06 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:01:06] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-12T22:01:07.112 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:06 smithi097 ceph-mon[24835]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:01:07.112 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:06 smithi097 ceph-mon[29620]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:01:07.113 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:01:06 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[59315]: level=info ts=2024-04-12T22:01:06.715Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000722277s 2024-04-12T22:01:07.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:06 smithi114 ceph-mon[25759]: from='client.14898 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:01:08.125 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:07 smithi114 ceph-mon[25759]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:01:08.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:07 smithi097 ceph-mon[24835]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:01:08.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:07 smithi097 ceph-mon[29620]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:01:08.494 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: 2024-04-12T22:01:08.123+0000 7f7e13cb8700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi097.xndkxy ... 2024-04-12T22:01:08.494 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: Creating ceph-iscsi config... 2024-04-12T22:01:08.494 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: Write file: /var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/iscsi.foo.smithi097.xndkxy/iscsi-gateway.cfg 2024-04-12T22:01:08.494 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: Write file: /var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/iscsi.foo.smithi097.xndkxy/tcmu-runner-entrypoint.sh 2024-04-12T22:01:08.494 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: Non-zero exit code 1 from systemctl restart ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy 2024-04-12T22:01:08.494 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: systemctl: stderr Job for ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service failed because the control process exited with error code. 2024-04-12T22:01:08.494 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: systemctl: stderr See "systemctl status ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service" and "journalctl -xeu ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service" for details. 2024-04-12T22:01:08.494 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: Traceback (most recent call last): 2024-04-12T22:01:08.494 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-12T22:01:08.494 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: return _run_code(code, main_globals, None, 2024-04-12T22:01:08.495 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-12T22:01:08.495 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: exec(code, run_globals) 2024-04-12T22:01:08.495 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 5534, in 2024-04-12T22:01:08.495 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 5522, in main 2024-04-12T22:01:08.495 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 3045, in command_deploy_from 2024-04-12T22:01:08.495 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 3080, in _common_deploy 2024-04-12T22:01:08.495 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 3100, in _deploy_daemon_container 2024-04-12T22:01:08.495 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 1072, in deploy_daemon 2024-04-12T22:01:08.495 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-12T22:01:08.496 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: RuntimeError: Failed command: systemctl restart ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy: Job for ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service failed because the control process exited with error code. 2024-04-12T22:01:08.496 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: See "systemctl status ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service" and "journalctl -xeu ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service" for details. 2024-04-12T22:01:08.496 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: Traceback (most recent call last): 2024-04-12T22:01:08.496 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-04-12T22:01:08.496 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-12T22:01:08.496 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2390, in _daemon_action 2024-04-12T22:01:08.496 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-12T22:01:08.496 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: File "/usr/share/ceph/mgr/cephadm/module.py", line 747, in wait_async 2024-04-12T22:01:08.496 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: return self.event_loop.get_result(coro, timeout) 2024-04-12T22:01:08.496 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-12T22:01:08.496 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: return future.result(timeout) 2024-04-12T22:01:08.496 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-12T22:01:08.496 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: return self.__get_result() 2024-04-12T22:01:08.496 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-12T22:01:08.497 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: raise self._exception 2024-04-12T22:01:08.497 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-04-12T22:01:08.497 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: config_blobs=daemon_spec.final_config, 2024-04-12T22:01:08.497 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-04-12T22:01:08.497 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-12T22:01:08.497 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi097.xndkxy ... 2024-04-12T22:01:08.497 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: Creating ceph-iscsi config... 2024-04-12T22:01:08.497 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: Write file: /var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/iscsi.foo.smithi097.xndkxy/iscsi-gateway.cfg 2024-04-12T22:01:08.497 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: Write file: /var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/iscsi.foo.smithi097.xndkxy/tcmu-runner-entrypoint.sh 2024-04-12T22:01:08.497 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: Non-zero exit code 1 from systemctl restart ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy 2024-04-12T22:01:08.497 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: systemctl: stderr Job for ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service failed because the control process exited with error code. 2024-04-12T22:01:08.497 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: systemctl: stderr See "systemctl status ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service" and "journalctl -xeu ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service" for details. 2024-04-12T22:01:08.497 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: Traceback (most recent call last): 2024-04-12T22:01:08.497 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-12T22:01:08.497 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: return _run_code(code, main_globals, None, 2024-04-12T22:01:08.498 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-12T22:01:08.498 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: exec(code, run_globals) 2024-04-12T22:01:08.498 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 5534, in 2024-04-12T22:01:08.498 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 5522, in main 2024-04-12T22:01:08.498 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 3045, in command_deploy_from 2024-04-12T22:01:08.498 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 3080, in _common_deploy 2024-04-12T22:01:08.498 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 3100, in _deploy_daemon_container 2024-04-12T22:01:08.498 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 1072, in deploy_daemon 2024-04-12T22:01:08.498 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-12T22:01:08.498 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: RuntimeError: Failed command: systemctl restart ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy: Job for ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service failed because the control process exited with error code. 2024-04-12T22:01:08.498 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: See "systemctl status ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service" and "journalctl -xeu ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service" for details. 2024-04-12T22:01:09.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:08 smithi097 ceph-mon[24835]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi097.xndkxy ... 2024-04-12T22:01:09.161 INFO:journalctl@ceph.mon.a.smithi097.stdout: Creating ceph-iscsi config... 2024-04-12T22:01:09.161 INFO:journalctl@ceph.mon.a.smithi097.stdout: Write file: /var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/iscsi.foo.smithi097.xndkxy/iscsi-gateway.cfg 2024-04-12T22:01:09.161 INFO:journalctl@ceph.mon.a.smithi097.stdout: Write file: /var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/iscsi.foo.smithi097.xndkxy/tcmu-runner-entrypoint.sh 2024-04-12T22:01:09.161 INFO:journalctl@ceph.mon.a.smithi097.stdout: Non-zero exit code 1 from systemctl restart ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy 2024-04-12T22:01:09.161 INFO:journalctl@ceph.mon.a.smithi097.stdout: systemctl: stderr Job for ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service failed because the control process exited with error code. 2024-04-12T22:01:09.161 INFO:journalctl@ceph.mon.a.smithi097.stdout: systemctl: stderr See "systemctl status ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service" and "journalctl -xeu ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service" for details. 2024-04-12T22:01:09.161 INFO:journalctl@ceph.mon.a.smithi097.stdout: Traceback (most recent call last): 2024-04-12T22:01:09.161 INFO:journalctl@ceph.mon.a.smithi097.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-12T22:01:09.162 INFO:journalctl@ceph.mon.a.smithi097.stdout: return _run_code(code, main_globals, None, 2024-04-12T22:01:09.162 INFO:journalctl@ceph.mon.a.smithi097.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-12T22:01:09.162 INFO:journalctl@ceph.mon.a.smithi097.stdout: exec(code, run_globals) 2024-04-12T22:01:09.162 INFO:journalctl@ceph.mon.a.smithi097.stdout: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 5534, in 2024-04-12T22:01:09.162 INFO:journalctl@ceph.mon.a.smithi097.stdout: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 5522, in main 2024-04-12T22:01:09.162 INFO:journalctl@ceph.mon.a.smithi097.stdout: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 3045, in command_deploy_from 2024-04-12T22:01:09.162 INFO:journalctl@ceph.mon.a.smithi097.stdout: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 3080, in _common_deploy 2024-04-12T22:01:09.162 INFO:journalctl@ceph.mon.a.smithi097.stdout: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 3100, in _deploy_daemon_container 2024-04-12T22:01:09.162 INFO:journalctl@ceph.mon.a.smithi097.stdout: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 1072, in deploy_daemon 2024-04-12T22:01:09.162 INFO:journalctl@ceph.mon.a.smithi097.stdout: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-12T22:01:09.162 INFO:journalctl@ceph.mon.a.smithi097.stdout: RuntimeError: Failed command: systemctl restart ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy: Job for ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service failed because the control process exited with error code. 2024-04-12T22:01:09.162 INFO:journalctl@ceph.mon.a.smithi097.stdout: See "systemctl status ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service" and "journalctl -xeu ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service" for details. 2024-04-12T22:01:09.162 INFO:journalctl@ceph.mon.a.smithi097.stdout: Traceback (most recent call last): 2024-04-12T22:01:09.163 INFO:journalctl@ceph.mon.a.smithi097.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-04-12T22:01:09.163 INFO:journalctl@ceph.mon.a.smithi097.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-12T22:01:09.163 INFO:journalctl@ceph.mon.a.smithi097.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2390, in _daemon_action 2024-04-12T22:01:09.163 INFO:journalctl@ceph.mon.a.smithi097.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-12T22:01:09.163 INFO:journalctl@ceph.mon.a.smithi097.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 747, in wait_async 2024-04-12T22:01:09.163 INFO:journalctl@ceph.mon.a.smithi097.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-12T22:01:09.163 INFO:journalctl@ceph.mon.a.smithi097.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-12T22:01:09.163 INFO:journalctl@ceph.mon.a.smithi097.stdout: return future.result(timeout) 2024-04-12T22:01:09.163 INFO:journalctl@ceph.mon.a.smithi097.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-12T22:01:09.163 INFO:journalctl@ceph.mon.a.smithi097.stdout: return self.__get_result() 2024-04-12T22:01:09.163 INFO:journalctl@ceph.mon.a.smithi097.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-12T22:01:09.163 INFO:journalctl@ceph.mon.a.smithi097.stdout: raise self._exception 2024-04-12T22:01:09.163 INFO:journalctl@ceph.mon.a.smithi097.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-04-12T22:01:09.163 INFO:journalctl@ceph.mon.a.smithi097.stdout: config_blobs=daemon_spec.final_config, 2024-04-12T22:01:09.164 INFO:journalctl@ceph.mon.a.smithi097.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-04-12T22:01:09.164 INFO:journalctl@ceph.mon.a.smithi097.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-12T22:01:09.164 INFO:journalctl@ceph.mon.a.smithi097.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi097.xndkxy ... 2024-04-12T22:01:09.164 INFO:journalctl@ceph.mon.a.smithi097.stdout: Creating ceph-iscsi config... 2024-04-12T22:01:09.164 INFO:journalctl@ceph.mon.a.smithi097.stdout: Write file: /var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/iscsi.foo.smithi097.xndkxy/iscsi-gateway.cfg 2024-04-12T22:01:09.164 INFO:journalctl@ceph.mon.a.smithi097.stdout: Write file: /var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/iscsi.foo.smithi097.xndkxy/tcmu-runner-entrypoint.sh 2024-04-12T22:01:09.164 INFO:journalctl@ceph.mon.a.smithi097.stdout: Non-zero exit code 1 from systemctl restart ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy 2024-04-12T22:01:09.164 INFO:journalctl@ceph.mon.a.smithi097.stdout: systemctl: stderr Job for ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service failed because the control process exited with error code. 2024-04-12T22:01:09.164 INFO:journalctl@ceph.mon.a.smithi097.stdout: systemctl: stderr See "systemctl status ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service" and "journalctl -xeu ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service" for details. 2024-04-12T22:01:09.164 INFO:journalctl@ceph.mon.a.smithi097.stdout: Traceback (most recent call last): 2024-04-12T22:01:09.164 INFO:journalctl@ceph.mon.a.smithi097.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-12T22:01:09.164 INFO:journalctl@ceph.mon.a.smithi097.stdout: return _run_code(code, main_globals, None, 2024-04-12T22:01:09.164 INFO:journalctl@ceph.mon.a.smithi097.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-12T22:01:09.164 INFO:journalctl@ceph.mon.a.smithi097.stdout: exec(code, run_globals) 2024-04-12T22:01:09.165 INFO:journalctl@ceph.mon.a.smithi097.stdout: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 5534, in 2024-04-12T22:01:09.165 INFO:journalctl@ceph.mon.a.smithi097.stdout: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 5522, in main 2024-04-12T22:01:09.165 INFO:journalctl@ceph.mon.a.smithi097.stdout: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 3045, in command_deploy_from 2024-04-12T22:01:09.165 INFO:journalctl@ceph.mon.a.smithi097.stdout: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 3080, in _common_deploy 2024-04-12T22:01:09.165 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:08 smithi097 ceph-mon[29620]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi097.xndkxy ... 2024-04-12T22:01:09.165 INFO:journalctl@ceph.mon.c.smithi097.stdout: Creating ceph-iscsi config... 2024-04-12T22:01:09.165 INFO:journalctl@ceph.mon.c.smithi097.stdout: Write file: /var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/iscsi.foo.smithi097.xndkxy/iscsi-gateway.cfg 2024-04-12T22:01:09.165 INFO:journalctl@ceph.mon.c.smithi097.stdout: Write file: /var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/iscsi.foo.smithi097.xndkxy/tcmu-runner-entrypoint.sh 2024-04-12T22:01:09.165 INFO:journalctl@ceph.mon.c.smithi097.stdout: Non-zero exit code 1 from systemctl restart ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy 2024-04-12T22:01:09.165 INFO:journalctl@ceph.mon.c.smithi097.stdout: systemctl: stderr Job for ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service failed because the control process exited with error code. 2024-04-12T22:01:09.166 INFO:journalctl@ceph.mon.c.smithi097.stdout: systemctl: stderr See "systemctl status ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service" and "journalctl -xeu ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service" for details. 2024-04-12T22:01:09.166 INFO:journalctl@ceph.mon.c.smithi097.stdout: Traceback (most recent call last): 2024-04-12T22:01:09.166 INFO:journalctl@ceph.mon.c.smithi097.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-12T22:01:09.166 INFO:journalctl@ceph.mon.c.smithi097.stdout: return _run_code(code, main_globals, None, 2024-04-12T22:01:09.166 INFO:journalctl@ceph.mon.c.smithi097.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-12T22:01:09.166 INFO:journalctl@ceph.mon.c.smithi097.stdout: exec(code, run_globals) 2024-04-12T22:01:09.166 INFO:journalctl@ceph.mon.c.smithi097.stdout: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 5534, in 2024-04-12T22:01:09.166 INFO:journalctl@ceph.mon.c.smithi097.stdout: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 5522, in main 2024-04-12T22:01:09.166 INFO:journalctl@ceph.mon.c.smithi097.stdout: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 3045, in command_deploy_from 2024-04-12T22:01:09.166 INFO:journalctl@ceph.mon.c.smithi097.stdout: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 3080, in _common_deploy 2024-04-12T22:01:09.166 INFO:journalctl@ceph.mon.c.smithi097.stdout: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 3100, in _deploy_daemon_container 2024-04-12T22:01:09.166 INFO:journalctl@ceph.mon.c.smithi097.stdout: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 1072, in deploy_daemon 2024-04-12T22:01:09.166 INFO:journalctl@ceph.mon.c.smithi097.stdout: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-12T22:01:09.167 INFO:journalctl@ceph.mon.c.smithi097.stdout: RuntimeError: Failed command: systemctl restart ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy: Job for ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service failed because the control process exited with error code. 2024-04-12T22:01:09.167 INFO:journalctl@ceph.mon.c.smithi097.stdout: See "systemctl status ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service" and "journalctl -xeu ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service" for details. 2024-04-12T22:01:09.167 INFO:journalctl@ceph.mon.c.smithi097.stdout: Traceback (most recent call last): 2024-04-12T22:01:09.167 INFO:journalctl@ceph.mon.c.smithi097.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-04-12T22:01:09.167 INFO:journalctl@ceph.mon.c.smithi097.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-12T22:01:09.167 INFO:journalctl@ceph.mon.c.smithi097.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2390, in _daemon_action 2024-04-12T22:01:09.167 INFO:journalctl@ceph.mon.c.smithi097.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-12T22:01:09.167 INFO:journalctl@ceph.mon.c.smithi097.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 747, in wait_async 2024-04-12T22:01:09.167 INFO:journalctl@ceph.mon.c.smithi097.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-12T22:01:09.167 INFO:journalctl@ceph.mon.c.smithi097.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-12T22:01:09.167 INFO:journalctl@ceph.mon.c.smithi097.stdout: return future.result(timeout) 2024-04-12T22:01:09.167 INFO:journalctl@ceph.mon.c.smithi097.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-12T22:01:09.167 INFO:journalctl@ceph.mon.c.smithi097.stdout: return self.__get_result() 2024-04-12T22:01:09.167 INFO:journalctl@ceph.mon.c.smithi097.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-12T22:01:09.167 INFO:journalctl@ceph.mon.c.smithi097.stdout: raise self._exception 2024-04-12T22:01:09.168 INFO:journalctl@ceph.mon.c.smithi097.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-04-12T22:01:09.168 INFO:journalctl@ceph.mon.c.smithi097.stdout: config_blobs=daemon_spec.final_config, 2024-04-12T22:01:09.168 INFO:journalctl@ceph.mon.c.smithi097.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-04-12T22:01:09.168 INFO:journalctl@ceph.mon.c.smithi097.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-12T22:01:09.168 INFO:journalctl@ceph.mon.c.smithi097.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi097.xndkxy ... 2024-04-12T22:01:09.168 INFO:journalctl@ceph.mon.c.smithi097.stdout: Creating ceph-iscsi config... 2024-04-12T22:01:09.168 INFO:journalctl@ceph.mon.c.smithi097.stdout: Write file: /var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/iscsi.foo.smithi097.xndkxy/iscsi-gateway.cfg 2024-04-12T22:01:09.168 INFO:journalctl@ceph.mon.c.smithi097.stdout: Write file: /var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/iscsi.foo.smithi097.xndkxy/tcmu-runner-entrypoint.sh 2024-04-12T22:01:09.168 INFO:journalctl@ceph.mon.c.smithi097.stdout: Non-zero exit code 1 from systemctl restart ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy 2024-04-12T22:01:09.168 INFO:journalctl@ceph.mon.c.smithi097.stdout: systemctl: stderr Job for ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service failed because the control process exited with error code. 2024-04-12T22:01:09.168 INFO:journalctl@ceph.mon.c.smithi097.stdout: systemctl: stderr See "systemctl status ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service" and "journalctl -xeu ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service" for details. 2024-04-12T22:01:09.168 INFO:journalctl@ceph.mon.c.smithi097.stdout: Traceback (most recent call last): 2024-04-12T22:01:09.168 INFO:journalctl@ceph.mon.c.smithi097.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-12T22:01:09.168 INFO:journalctl@ceph.mon.c.smithi097.stdout: return _run_code(code, main_globals, None, 2024-04-12T22:01:09.169 INFO:journalctl@ceph.mon.c.smithi097.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-12T22:01:09.169 INFO:journalctl@ceph.mon.c.smithi097.stdout: exec(code, run_globals) 2024-04-12T22:01:09.169 INFO:journalctl@ceph.mon.c.smithi097.stdout: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 5534, in 2024-04-12T22:01:09.169 INFO:journalctl@ceph.mon.c.smithi097.stdout: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 5522, in main 2024-04-12T22:01:09.169 INFO:journalctl@ceph.mon.c.smithi097.stdout: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 3045, in command_deploy_from 2024-04-12T22:01:09.169 INFO:journalctl@ceph.mon.c.smithi097.stdout: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 3080, in _common_deploy 2024-04-12T22:01:09.169 INFO:journalctl@ceph.mon.a.smithi097.stdout: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 3100, in _deploy_daemon_container 2024-04-12T22:01:09.169 INFO:journalctl@ceph.mon.a.smithi097.stdout: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 1072, in deploy_daemon 2024-04-12T22:01:09.169 INFO:journalctl@ceph.mon.a.smithi097.stdout: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-12T22:01:09.169 INFO:journalctl@ceph.mon.a.smithi097.stdout: RuntimeError: Failed command: systemctl restart ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy: Job for ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service failed because the control process exited with error code. 2024-04-12T22:01:09.169 INFO:journalctl@ceph.mon.a.smithi097.stdout: See "systemctl status ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service" and "journalctl -xeu ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service" for details. 2024-04-12T22:01:09.169 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:08 smithi097 ceph-mon[24835]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-12T22:01:09.170 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:08 smithi097 ceph-mon[24835]: Reconfiguring daemon prometheus.a on smithi114 2024-04-12T22:01:09.170 INFO:journalctl@ceph.mon.c.smithi097.stdout: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 3100, in _deploy_daemon_container 2024-04-12T22:01:09.170 INFO:journalctl@ceph.mon.c.smithi097.stdout: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 1072, in deploy_daemon 2024-04-12T22:01:09.170 INFO:journalctl@ceph.mon.c.smithi097.stdout: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-12T22:01:09.170 INFO:journalctl@ceph.mon.c.smithi097.stdout: RuntimeError: Failed command: systemctl restart ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy: Job for ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service failed because the control process exited with error code. 2024-04-12T22:01:09.170 INFO:journalctl@ceph.mon.c.smithi097.stdout: See "systemctl status ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service" and "journalctl -xeu ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service" for details. 2024-04-12T22:01:09.170 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:08 smithi097 ceph-mon[29620]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-12T22:01:09.170 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:08 smithi097 ceph-mon[29620]: Reconfiguring daemon prometheus.a on smithi114 2024-04-12T22:01:09.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-mon[25759]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi097.xndkxy ... 2024-04-12T22:01:09.244 INFO:journalctl@ceph.mon.b.smithi114.stdout: Creating ceph-iscsi config... 2024-04-12T22:01:09.244 INFO:journalctl@ceph.mon.b.smithi114.stdout: Write file: /var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/iscsi.foo.smithi097.xndkxy/iscsi-gateway.cfg 2024-04-12T22:01:09.244 INFO:journalctl@ceph.mon.b.smithi114.stdout: Write file: /var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/iscsi.foo.smithi097.xndkxy/tcmu-runner-entrypoint.sh 2024-04-12T22:01:09.244 INFO:journalctl@ceph.mon.b.smithi114.stdout: Non-zero exit code 1 from systemctl restart ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy 2024-04-12T22:01:09.244 INFO:journalctl@ceph.mon.b.smithi114.stdout: systemctl: stderr Job for ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service failed because the control process exited with error code. 2024-04-12T22:01:09.244 INFO:journalctl@ceph.mon.b.smithi114.stdout: systemctl: stderr See "systemctl status ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service" and "journalctl -xeu ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service" for details. 2024-04-12T22:01:09.244 INFO:journalctl@ceph.mon.b.smithi114.stdout: Traceback (most recent call last): 2024-04-12T22:01:09.245 INFO:journalctl@ceph.mon.b.smithi114.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-12T22:01:09.245 INFO:journalctl@ceph.mon.b.smithi114.stdout: return _run_code(code, main_globals, None, 2024-04-12T22:01:09.245 INFO:journalctl@ceph.mon.b.smithi114.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-12T22:01:09.245 INFO:journalctl@ceph.mon.b.smithi114.stdout: exec(code, run_globals) 2024-04-12T22:01:09.245 INFO:journalctl@ceph.mon.b.smithi114.stdout: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 5534, in 2024-04-12T22:01:09.245 INFO:journalctl@ceph.mon.b.smithi114.stdout: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 5522, in main 2024-04-12T22:01:09.245 INFO:journalctl@ceph.mon.b.smithi114.stdout: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 3045, in command_deploy_from 2024-04-12T22:01:09.245 INFO:journalctl@ceph.mon.b.smithi114.stdout: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 3080, in _common_deploy 2024-04-12T22:01:09.245 INFO:journalctl@ceph.mon.b.smithi114.stdout: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 3100, in _deploy_daemon_container 2024-04-12T22:01:09.245 INFO:journalctl@ceph.mon.b.smithi114.stdout: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 1072, in deploy_daemon 2024-04-12T22:01:09.245 INFO:journalctl@ceph.mon.b.smithi114.stdout: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-12T22:01:09.245 INFO:journalctl@ceph.mon.b.smithi114.stdout: RuntimeError: Failed command: systemctl restart ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy: Job for ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service failed because the control process exited with error code. 2024-04-12T22:01:09.245 INFO:journalctl@ceph.mon.b.smithi114.stdout: See "systemctl status ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service" and "journalctl -xeu ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service" for details. 2024-04-12T22:01:09.245 INFO:journalctl@ceph.mon.b.smithi114.stdout: Traceback (most recent call last): 2024-04-12T22:01:09.245 INFO:journalctl@ceph.mon.b.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-04-12T22:01:09.246 INFO:journalctl@ceph.mon.b.smithi114.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-12T22:01:09.246 INFO:journalctl@ceph.mon.b.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2390, in _daemon_action 2024-04-12T22:01:09.246 INFO:journalctl@ceph.mon.b.smithi114.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-12T22:01:09.246 INFO:journalctl@ceph.mon.b.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 747, in wait_async 2024-04-12T22:01:09.246 INFO:journalctl@ceph.mon.b.smithi114.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-12T22:01:09.246 INFO:journalctl@ceph.mon.b.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-12T22:01:09.246 INFO:journalctl@ceph.mon.b.smithi114.stdout: return future.result(timeout) 2024-04-12T22:01:09.246 INFO:journalctl@ceph.mon.b.smithi114.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-12T22:01:09.246 INFO:journalctl@ceph.mon.b.smithi114.stdout: return self.__get_result() 2024-04-12T22:01:09.246 INFO:journalctl@ceph.mon.b.smithi114.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-12T22:01:09.246 INFO:journalctl@ceph.mon.b.smithi114.stdout: raise self._exception 2024-04-12T22:01:09.247 INFO:journalctl@ceph.mon.b.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-04-12T22:01:09.247 INFO:journalctl@ceph.mon.b.smithi114.stdout: config_blobs=daemon_spec.final_config, 2024-04-12T22:01:09.247 INFO:journalctl@ceph.mon.b.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-04-12T22:01:09.247 INFO:journalctl@ceph.mon.b.smithi114.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-12T22:01:09.247 INFO:journalctl@ceph.mon.b.smithi114.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi097.xndkxy ... 2024-04-12T22:01:09.247 INFO:journalctl@ceph.mon.b.smithi114.stdout: Creating ceph-iscsi config... 2024-04-12T22:01:09.247 INFO:journalctl@ceph.mon.b.smithi114.stdout: Write file: /var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/iscsi.foo.smithi097.xndkxy/iscsi-gateway.cfg 2024-04-12T22:01:09.247 INFO:journalctl@ceph.mon.b.smithi114.stdout: Write file: /var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/iscsi.foo.smithi097.xndkxy/tcmu-runner-entrypoint.sh 2024-04-12T22:01:09.247 INFO:journalctl@ceph.mon.b.smithi114.stdout: Non-zero exit code 1 from systemctl restart ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy 2024-04-12T22:01:09.247 INFO:journalctl@ceph.mon.b.smithi114.stdout: systemctl: stderr Job for ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service failed because the control process exited with error code. 2024-04-12T22:01:09.247 INFO:journalctl@ceph.mon.b.smithi114.stdout: systemctl: stderr See "systemctl status ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service" and "journalctl -xeu ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service" for details. 2024-04-12T22:01:09.247 INFO:journalctl@ceph.mon.b.smithi114.stdout: Traceback (most recent call last): 2024-04-12T22:01:09.247 INFO:journalctl@ceph.mon.b.smithi114.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-12T22:01:09.247 INFO:journalctl@ceph.mon.b.smithi114.stdout: return _run_code(code, main_globals, None, 2024-04-12T22:01:09.248 INFO:journalctl@ceph.mon.b.smithi114.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-12T22:01:09.248 INFO:journalctl@ceph.mon.b.smithi114.stdout: exec(code, run_globals) 2024-04-12T22:01:09.248 INFO:journalctl@ceph.mon.b.smithi114.stdout: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 5534, in 2024-04-12T22:01:09.248 INFO:journalctl@ceph.mon.b.smithi114.stdout: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 5522, in main 2024-04-12T22:01:09.248 INFO:journalctl@ceph.mon.b.smithi114.stdout: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 3045, in command_deploy_from 2024-04-12T22:01:09.248 INFO:journalctl@ceph.mon.b.smithi114.stdout: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 3080, in _common_deploy 2024-04-12T22:01:09.248 INFO:journalctl@ceph.mon.b.smithi114.stdout: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 3100, in _deploy_daemon_container 2024-04-12T22:01:09.248 INFO:journalctl@ceph.mon.b.smithi114.stdout: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 1072, in deploy_daemon 2024-04-12T22:01:09.248 INFO:journalctl@ceph.mon.b.smithi114.stdout: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-12T22:01:09.248 INFO:journalctl@ceph.mon.b.smithi114.stdout: RuntimeError: Failed command: systemctl restart ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy: Job for ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service failed because the control process exited with error code. 2024-04-12T22:01:09.248 INFO:journalctl@ceph.mon.b.smithi114.stdout: See "systemctl status ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service" and "journalctl -xeu ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service" for details. 2024-04-12T22:01:09.248 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-mon[25759]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-12T22:01:09.248 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:08 smithi114 ceph-mon[25759]: Reconfiguring daemon prometheus.a on smithi114 2024-04-12T22:01:10.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:09 smithi097 ceph-mon[24835]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:01:10.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:09 smithi097 ceph-mon[29620]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:01:10.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:09 smithi114 ceph-mon[25759]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:01:12.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:12 smithi097 ceph-mon[24835]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-04-12T22:01:12.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:12 smithi097 ceph-mon[29620]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-04-12T22:01:12.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:12 smithi114 ceph-mon[25759]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-04-12T22:01:12.493 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:01:12 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[42649]: ts=2024-04-12T22:01:12.314Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.97:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.114:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-12T22:01:14.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:14 smithi097 ceph-mon[24835]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-04-12T22:01:14.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:14 smithi097 ceph-mon[29620]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-04-12T22:01:14.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:14 smithi114 ceph-mon[25759]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-04-12T22:01:14.493 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:14 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:01:14] "GET /metrics HTTP/1.1" 200 38147 "" "Prometheus/2.33.4" 2024-04-12T22:01:15.135 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:01:14 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[59315]: level=info ts=2024-04-12T22:01:14.718Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.003739912s 2024-04-12T22:01:15.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:15 smithi097 ceph-mon[24835]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:01:15.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:15 smithi097 ceph-mon[29620]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:01:15.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:15 smithi114 ceph-mon[25759]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:01:16.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:16 smithi097 ceph-mon[24835]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-04-12T22:01:16.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:16 smithi097 ceph-mon[29620]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-04-12T22:01:16.411 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:01:16 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[25068]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:01:16] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-12T22:01:16.447 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:16 smithi114 ceph-mon[25759]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-04-12T22:01:18.400 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:18 smithi097 ceph-mon[24835]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-04-12T22:01:18.400 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:18 smithi097 ceph-mon[29620]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-04-12T22:01:18.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:18 smithi114 ceph-mon[25759]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-04-12T22:01:18.821 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:01:18 smithi114 systemd[1]: Stopping Ceph prometheus.a for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:01:18.822 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:01:18 smithi114 bash[50419]: Error: no container with name or ID "ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus.a" found: no such container 2024-04-12T22:01:19.106 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:01:18 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[42649]: ts=2024-04-12T22:01:18.820Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-04-12T22:01:19.106 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:01:18 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[42649]: ts=2024-04-12T22:01:18.820Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-04-12T22:01:19.107 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:01:18 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[42649]: ts=2024-04-12T22:01:18.820Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-04-12T22:01:19.107 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:01:18 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[42649]: ts=2024-04-12T22:01:18.820Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-04-12T22:01:19.107 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:01:18 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[42649]: ts=2024-04-12T22:01:18.820Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-04-12T22:01:19.107 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:01:18 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[42649]: ts=2024-04-12T22:01:18.820Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-04-12T22:01:19.107 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:01:18 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[42649]: ts=2024-04-12T22:01:18.820Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-04-12T22:01:19.107 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:01:18 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[42649]: ts=2024-04-12T22:01:18.820Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-04-12T22:01:19.107 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:01:18 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[42649]: ts=2024-04-12T22:01:18.820Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-04-12T22:01:19.107 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:01:18 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[42649]: ts=2024-04-12T22:01:18.822Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-04-12T22:01:19.107 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:01:18 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[42649]: ts=2024-04-12T22:01:18.822Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-04-12T22:01:19.107 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:01:18 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[42649]: ts=2024-04-12T22:01:18.822Z caller=main.go:1066 level=info msg="See you next time!" 2024-04-12T22:01:19.107 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:01:18 smithi114 podman[50430]: 2024-04-12 22:01:18.830929589 +0000 UTC m=+0.125813901 container died 7a2b6dbe5d8b26479ac240c59da735275ab236572bc449f90775450eb416b4f9 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-12T22:01:19.453 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:01:19 smithi114 podman[50430]: 2024-04-12 22:01:19.21224427 +0000 UTC m=+0.507128581 container cleanup 7a2b6dbe5d8b26479ac240c59da735275ab236572bc449f90775450eb416b4f9 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-12T22:01:19.453 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:01:19 smithi114 bash[50430]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a 2024-04-12T22:01:19.453 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:01:19 smithi114 bash[50457]: Error: no container with name or ID "ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus.a" found: no such container 2024-04-12T22:01:19.723 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:19 smithi114 ceph-mon[25759]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-12T22:01:19.723 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:01:19 smithi114 podman[50444]: 2024-04-12 22:01:19.463434704 +0000 UTC m=+0.633733238 container remove 7a2b6dbe5d8b26479ac240c59da735275ab236572bc449f90775450eb416b4f9 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-12T22:01:19.723 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:01:19 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-04-12T22:01:19.723 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:01:19 smithi114 systemd[1]: Stopped Ceph prometheus.a for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:01:19.723 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:01:19 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@prometheus.a.service: Consumed 1.486s CPU time. 2024-04-12T22:01:19.723 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:01:19 smithi114 systemd[1]: Starting Ceph prometheus.a for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:01:19.723 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:01:19 smithi114 podman[50531]: 2024-04-12 22:01:19.597423383 +0000 UTC m=+0.018971072 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-04-12T22:01:19.724 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:01:19 smithi114 podman[50531]: 2024-04-12 22:01:19.722668239 +0000 UTC m=+0.144215930 container create e98d7a0f3c0cb95c1996341a21c0dada1e11094514b7c4a217cf14ef6074317a (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-12T22:01:19.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:19 smithi097 ceph-mon[24835]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-12T22:01:19.910 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:19 smithi097 ceph-mon[29620]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-12T22:01:19.974 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:01:19 smithi114 podman[50531]: 2024-04-12 22:01:19.941859145 +0000 UTC m=+0.363406835 container init e98d7a0f3c0cb95c1996341a21c0dada1e11094514b7c4a217cf14ef6074317a (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-12T22:01:19.974 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:01:19 smithi114 podman[50531]: 2024-04-12 22:01:19.944287067 +0000 UTC m=+0.365834764 container start e98d7a0f3c0cb95c1996341a21c0dada1e11094514b7c4a217cf14ef6074317a (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-12T22:01:19.974 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:01:19 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[50546]: ts=2024-04-12T22:01:19.968Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-04-12T22:01:19.975 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:01:19 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[50546]: ts=2024-04-12T22:01:19.968Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-04-12T22:01:19.975 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:01:19 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[50546]: ts=2024-04-12T22:01:19.968Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-04-12T22:01:19.975 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:01:19 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[50546]: ts=2024-04-12T22:01:19.968Z caller=main.go:518 level=info host_details="(Linux 5.14.0-435.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024 x86_64 smithi114 (none))" 2024-04-12T22:01:19.975 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:01:19 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[50546]: ts=2024-04-12T22:01:19.968Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-04-12T22:01:19.975 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:01:19 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[50546]: ts=2024-04-12T22:01:19.968Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-04-12T22:01:19.975 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:01:19 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[50546]: ts=2024-04-12T22:01:19.969Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-04-12T22:01:19.975 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:01:19 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[50546]: ts=2024-04-12T22:01:19.970Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-04-12T22:01:19.975 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:01:19 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[50546]: ts=2024-04-12T22:01:19.971Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-04-12T22:01:19.975 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:01:19 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[50546]: ts=2024-04-12T22:01:19.973Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-04-12T22:01:20.243 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:01:19 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[50546]: ts=2024-04-12T22:01:19.975Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.560083ms 2024-04-12T22:01:20.244 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:01:19 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[50546]: ts=2024-04-12T22:01:19.975Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-04-12T22:01:20.244 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:01:20 smithi114 bash[50531]: e98d7a0f3c0cb95c1996341a21c0dada1e11094514b7c4a217cf14ef6074317a 2024-04-12T22:01:20.244 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:01:20 smithi114 systemd[1]: Started Ceph prometheus.a for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:01:20.244 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:01:20 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[50546]: ts=2024-04-12T22:01:20.184Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=2 2024-04-12T22:01:20.669 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:20 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/4243108360' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-12T22:01:20.669 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:20 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/2721259191' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6801/560924382"}]: dispatch 2024-04-12T22:01:20.669 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:20 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:20.669 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:20 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:20.670 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:20 smithi114 ceph-mon[25759]: Reconfiguring node-exporter.b (dependencies changed)... 2024-04-12T22:01:20.670 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:20 smithi114 ceph-mon[25759]: Reconfiguring daemon node-exporter.b on smithi114 2024-04-12T22:01:20.670 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:01:20 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[50546]: ts=2024-04-12T22:01:20.446Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=2 2024-04-12T22:01:20.670 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:01:20 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[50546]: ts=2024-04-12T22:01:20.446Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=2 2024-04-12T22:01:20.670 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:01:20 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[50546]: ts=2024-04-12T22:01:20.446Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=48.31µs wal_replay_duration=471.498249ms total_replay_duration=473.132566ms 2024-04-12T22:01:20.670 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:01:20 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[50546]: ts=2024-04-12T22:01:20.452Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-04-12T22:01:20.670 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:01:20 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[50546]: ts=2024-04-12T22:01:20.452Z caller=main.go:947 level=info msg="TSDB started" 2024-04-12T22:01:20.670 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:01:20 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[50546]: ts=2024-04-12T22:01:20.452Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-04-12T22:01:20.670 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:01:20 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[50546]: ts=2024-04-12T22:01:20.480Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=28.805044ms db_storage=920ns remote_storage=1.76µs web_handler=371ns query_engine=1.12µs scrape=531.222µs scrape_sd=147.596µs notify=17.126µs notify_sd=15.404µs rules=27.730949ms 2024-04-12T22:01:20.670 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:01:20 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[50546]: ts=2024-04-12T22:01:20.480Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-04-12T22:01:20.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:20 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/4243108360' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-12T22:01:20.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:20 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/2721259191' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6801/560924382"}]: dispatch 2024-04-12T22:01:20.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:20 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:20.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:20 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:20.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:20 smithi097 ceph-mon[24835]: Reconfiguring node-exporter.b (dependencies changed)... 2024-04-12T22:01:20.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:20 smithi097 ceph-mon[24835]: Reconfiguring daemon node-exporter.b on smithi114 2024-04-12T22:01:20.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:20 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/4243108360' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-12T22:01:20.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:20 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/2721259191' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6801/560924382"}]: dispatch 2024-04-12T22:01:20.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:20 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:20.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:20 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:20.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:20 smithi097 ceph-mon[29620]: Reconfiguring node-exporter.b (dependencies changed)... 2024-04-12T22:01:20.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:20 smithi097 ceph-mon[29620]: Reconfiguring daemon node-exporter.b on smithi114 2024-04-12T22:01:20.993 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:20 smithi114 systemd[1]: Stopping Ceph node-exporter.b for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:01:20.993 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:20 smithi114 bash[50666]: Error: no container with name or ID "ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter.b" found: no such container 2024-04-12T22:01:20.993 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:20 smithi114 podman[50679]: 2024-04-12 22:01:20.848052306 +0000 UTC m=+0.142903230 container died 20d4da15891ea70628d2dfc691b675971db256218e5c1f1f8f3314c945a078f1 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-12T22:01:21.462 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 podman[50679]: 2024-04-12 22:01:21.35424816 +0000 UTC m=+0.649099093 container cleanup 20d4da15891ea70628d2dfc691b675971db256218e5c1f1f8f3314c945a078f1 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-12T22:01:21.462 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 bash[50679]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b 2024-04-12T22:01:21.463 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 bash[50709]: Error: no container with name or ID "ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter.b" found: no such container 2024-04-12T22:01:21.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:21 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/2721259191' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6801/560924382"}]': finished 2024-04-12T22:01:21.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:21 smithi114 ceph-mon[25759]: osdmap e98: 8 total, 8 up, 8 in 2024-04-12T22:01:21.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:21 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/2262972124' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6800/560924382"}]: dispatch 2024-04-12T22:01:21.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:21 smithi114 ceph-mon[25759]: from='client.? ' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6800/560924382"}]: dispatch 2024-04-12T22:01:21.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:21 smithi114 ceph-mon[25759]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-04-12T22:01:21.744 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 podman[50695]: 2024-04-12 22:01:21.547189633 +0000 UTC m=+0.703781400 container remove 20d4da15891ea70628d2dfc691b675971db256218e5c1f1f8f3314c945a078f1 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-12T22:01:21.744 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-04-12T22:01:21.744 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-04-12T22:01:21.744 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 systemd[1]: Stopped Ceph node-exporter.b for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:01:21.744 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@node-exporter.b.service: Consumed 2.817s CPU time. 2024-04-12T22:01:21.744 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 systemd[1]: Starting Ceph node-exporter.b for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:01:21.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:21 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/2721259191' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6801/560924382"}]': finished 2024-04-12T22:01:21.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:21 smithi097 ceph-mon[24835]: osdmap e98: 8 total, 8 up, 8 in 2024-04-12T22:01:21.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:21 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/2262972124' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6800/560924382"}]: dispatch 2024-04-12T22:01:21.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:21 smithi097 ceph-mon[24835]: from='client.? ' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6800/560924382"}]: dispatch 2024-04-12T22:01:21.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:21 smithi097 ceph-mon[24835]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-04-12T22:01:21.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:21 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/2721259191' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6801/560924382"}]': finished 2024-04-12T22:01:21.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:21 smithi097 ceph-mon[29620]: osdmap e98: 8 total, 8 up, 8 in 2024-04-12T22:01:21.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:21 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/2262972124' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6800/560924382"}]: dispatch 2024-04-12T22:01:21.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:21 smithi097 ceph-mon[29620]: from='client.? ' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6800/560924382"}]: dispatch 2024-04-12T22:01:21.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:21 smithi097 ceph-mon[29620]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-04-12T22:01:22.051 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 podman[50777]: 2024-04-12 22:01:21.673105893 +0000 UTC m=+0.017346570 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-04-12T22:01:22.051 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 podman[50777]: 2024-04-12 22:01:21.773241182 +0000 UTC m=+0.117481856 container create 904afc45dc2f5bb072538cbc23d7be546f41eaf0a22a24eda03f0250ce8ead29 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-12T22:01:22.052 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 podman[50777]: 2024-04-12 22:01:21.967156502 +0000 UTC m=+0.311397166 container init 904afc45dc2f5bb072538cbc23d7be546f41eaf0a22a24eda03f0250ce8ead29 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-12T22:01:22.052 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 podman[50777]: 2024-04-12 22:01:21.969563651 +0000 UTC m=+0.313804317 container start 904afc45dc2f5bb072538cbc23d7be546f41eaf0a22a24eda03f0250ce8ead29 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-12T22:01:22.052 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[50792]: ts=2024-04-12T22:01:21.972Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-04-12T22:01:22.052 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[50792]: ts=2024-04-12T22:01:21.972Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-04-12T22:01:22.052 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[50792]: ts=2024-04-12T22:01:21.973Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-04-12T22:01:22.052 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[50792]: ts=2024-04-12T22:01:21.973Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-04-12T22:01:22.052 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[50792]: ts=2024-04-12T22:01:21.973Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-04-12T22:01:22.052 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[50792]: ts=2024-04-12T22:01:21.973Z caller=node_exporter.go:115 level=info collector=arp 2024-04-12T22:01:22.052 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[50792]: ts=2024-04-12T22:01:21.973Z caller=node_exporter.go:115 level=info collector=bcache 2024-04-12T22:01:22.052 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[50792]: ts=2024-04-12T22:01:21.973Z caller=node_exporter.go:115 level=info collector=bonding 2024-04-12T22:01:22.052 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[50792]: ts=2024-04-12T22:01:21.973Z caller=node_exporter.go:115 level=info collector=btrfs 2024-04-12T22:01:22.052 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[50792]: ts=2024-04-12T22:01:21.973Z caller=node_exporter.go:115 level=info collector=conntrack 2024-04-12T22:01:22.053 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[50792]: ts=2024-04-12T22:01:21.973Z caller=node_exporter.go:115 level=info collector=cpu 2024-04-12T22:01:22.053 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[50792]: ts=2024-04-12T22:01:21.973Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-04-12T22:01:22.054 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[50792]: ts=2024-04-12T22:01:21.973Z caller=node_exporter.go:115 level=info collector=diskstats 2024-04-12T22:01:22.054 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[50792]: ts=2024-04-12T22:01:21.973Z caller=node_exporter.go:115 level=info collector=dmi 2024-04-12T22:01:22.054 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[50792]: ts=2024-04-12T22:01:21.973Z caller=node_exporter.go:115 level=info collector=edac 2024-04-12T22:01:22.054 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[50792]: ts=2024-04-12T22:01:21.973Z caller=node_exporter.go:115 level=info collector=entropy 2024-04-12T22:01:22.054 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[50792]: ts=2024-04-12T22:01:21.973Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-04-12T22:01:22.054 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[50792]: ts=2024-04-12T22:01:21.973Z caller=node_exporter.go:115 level=info collector=filefd 2024-04-12T22:01:22.054 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[50792]: ts=2024-04-12T22:01:21.973Z caller=node_exporter.go:115 level=info collector=filesystem 2024-04-12T22:01:22.055 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[50792]: ts=2024-04-12T22:01:21.973Z caller=node_exporter.go:115 level=info collector=hwmon 2024-04-12T22:01:22.055 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[50792]: ts=2024-04-12T22:01:21.973Z caller=node_exporter.go:115 level=info collector=infiniband 2024-04-12T22:01:22.055 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[50792]: ts=2024-04-12T22:01:21.973Z caller=node_exporter.go:115 level=info collector=ipvs 2024-04-12T22:01:22.055 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[50792]: ts=2024-04-12T22:01:21.973Z caller=node_exporter.go:115 level=info collector=loadavg 2024-04-12T22:01:22.055 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[50792]: ts=2024-04-12T22:01:21.973Z caller=node_exporter.go:115 level=info collector=mdadm 2024-04-12T22:01:22.055 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[50792]: ts=2024-04-12T22:01:21.973Z caller=node_exporter.go:115 level=info collector=meminfo 2024-04-12T22:01:22.055 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[50792]: ts=2024-04-12T22:01:21.973Z caller=node_exporter.go:115 level=info collector=netclass 2024-04-12T22:01:22.055 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[50792]: ts=2024-04-12T22:01:21.973Z caller=node_exporter.go:115 level=info collector=netdev 2024-04-12T22:01:22.055 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[50792]: ts=2024-04-12T22:01:21.973Z caller=node_exporter.go:115 level=info collector=netstat 2024-04-12T22:01:22.055 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[50792]: ts=2024-04-12T22:01:21.973Z caller=node_exporter.go:115 level=info collector=nfs 2024-04-12T22:01:22.055 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[50792]: ts=2024-04-12T22:01:21.973Z caller=node_exporter.go:115 level=info collector=nfsd 2024-04-12T22:01:22.055 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[50792]: ts=2024-04-12T22:01:21.973Z caller=node_exporter.go:115 level=info collector=nvme 2024-04-12T22:01:22.055 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[50792]: ts=2024-04-12T22:01:21.973Z caller=node_exporter.go:115 level=info collector=os 2024-04-12T22:01:22.055 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[50792]: ts=2024-04-12T22:01:21.973Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-04-12T22:01:22.056 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[50792]: ts=2024-04-12T22:01:21.973Z caller=node_exporter.go:115 level=info collector=pressure 2024-04-12T22:01:22.056 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[50792]: ts=2024-04-12T22:01:21.973Z caller=node_exporter.go:115 level=info collector=rapl 2024-04-12T22:01:22.056 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[50792]: ts=2024-04-12T22:01:21.973Z caller=node_exporter.go:115 level=info collector=schedstat 2024-04-12T22:01:22.056 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[50792]: ts=2024-04-12T22:01:21.973Z caller=node_exporter.go:115 level=info collector=sockstat 2024-04-12T22:01:22.056 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[50792]: ts=2024-04-12T22:01:21.973Z caller=node_exporter.go:115 level=info collector=softnet 2024-04-12T22:01:22.056 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[50792]: ts=2024-04-12T22:01:21.973Z caller=node_exporter.go:115 level=info collector=stat 2024-04-12T22:01:22.056 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[50792]: ts=2024-04-12T22:01:21.973Z caller=node_exporter.go:115 level=info collector=tapestats 2024-04-12T22:01:22.056 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[50792]: ts=2024-04-12T22:01:21.973Z caller=node_exporter.go:115 level=info collector=textfile 2024-04-12T22:01:22.056 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[50792]: ts=2024-04-12T22:01:21.973Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-04-12T22:01:22.056 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[50792]: ts=2024-04-12T22:01:21.973Z caller=node_exporter.go:115 level=info collector=time 2024-04-12T22:01:22.056 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[50792]: ts=2024-04-12T22:01:21.973Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-04-12T22:01:22.056 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[50792]: ts=2024-04-12T22:01:21.973Z caller=node_exporter.go:115 level=info collector=uname 2024-04-12T22:01:22.056 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[50792]: ts=2024-04-12T22:01:21.973Z caller=node_exporter.go:115 level=info collector=vmstat 2024-04-12T22:01:22.056 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[50792]: ts=2024-04-12T22:01:21.973Z caller=node_exporter.go:115 level=info collector=xfs 2024-04-12T22:01:22.056 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[50792]: ts=2024-04-12T22:01:21.974Z caller=node_exporter.go:115 level=info collector=zfs 2024-04-12T22:01:22.056 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[50792]: ts=2024-04-12T22:01:21.974Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-04-12T22:01:22.056 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b[50792]: ts=2024-04-12T22:01:21.974Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-04-12T22:01:22.057 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:22 smithi114 bash[50777]: 904afc45dc2f5bb072538cbc23d7be546f41eaf0a22a24eda03f0250ce8ead29 2024-04-12T22:01:22.470 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:01:22 smithi114 systemd[1]: Started Ceph node-exporter.b for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:01:22.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:22 smithi114 ceph-mon[25759]: from='client.? ' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6800/560924382"}]': finished 2024-04-12T22:01:22.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:22 smithi114 ceph-mon[25759]: osdmap e99: 8 total, 8 up, 8 in 2024-04-12T22:01:22.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:22 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/2391241910' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/1069516662"}]: dispatch 2024-04-12T22:01:22.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:22 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:22.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:22 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:22.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:22 smithi114 ceph-mon[25759]: Reconfiguring grafana.a (dependencies changed)... 2024-04-12T22:01:22.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:22 smithi114 ceph-mon[25759]: Regenerating cephadm self-signed grafana TLS certificates 2024-04-12T22:01:22.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:22 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:22.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:22 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:22.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:22 smithi114 ceph-mon[25759]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-12T22:01:22.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:22 smithi114 ceph-mon[25759]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-12T22:01:22.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:22 smithi114 ceph-mon[25759]: Reconfiguring daemon grafana.a on smithi114 2024-04-12T22:01:22.910 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:22 smithi097 ceph-mon[29620]: from='client.? ' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6800/560924382"}]': finished 2024-04-12T22:01:22.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:22 smithi097 ceph-mon[29620]: osdmap e99: 8 total, 8 up, 8 in 2024-04-12T22:01:22.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:22 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/2391241910' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/1069516662"}]: dispatch 2024-04-12T22:01:22.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:22 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:22.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:22 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:22.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:22 smithi097 ceph-mon[29620]: Reconfiguring grafana.a (dependencies changed)... 2024-04-12T22:01:22.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:22 smithi097 ceph-mon[29620]: Regenerating cephadm self-signed grafana TLS certificates 2024-04-12T22:01:22.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:22 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:22.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:22 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:22.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:22 smithi097 ceph-mon[29620]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-12T22:01:22.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:22 smithi097 ceph-mon[29620]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-12T22:01:22.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:22 smithi097 ceph-mon[29620]: Reconfiguring daemon grafana.a on smithi114 2024-04-12T22:01:22.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:22 smithi097 ceph-mon[24835]: from='client.? ' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6800/560924382"}]': finished 2024-04-12T22:01:22.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:22 smithi097 ceph-mon[24835]: osdmap e99: 8 total, 8 up, 8 in 2024-04-12T22:01:22.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:22 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/2391241910' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/1069516662"}]: dispatch 2024-04-12T22:01:22.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:22 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:22.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:22 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:22.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:22 smithi097 ceph-mon[24835]: Reconfiguring grafana.a (dependencies changed)... 2024-04-12T22:01:22.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:22 smithi097 ceph-mon[24835]: Regenerating cephadm self-signed grafana TLS certificates 2024-04-12T22:01:22.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:22 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:22.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:22 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:22.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:22 smithi097 ceph-mon[24835]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-12T22:01:22.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:22 smithi097 ceph-mon[24835]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-12T22:01:22.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:22 smithi097 ceph-mon[24835]: Reconfiguring daemon grafana.a on smithi114 2024-04-12T22:01:23.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:23 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/2391241910' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/1069516662"}]': finished 2024-04-12T22:01:23.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:23 smithi114 ceph-mon[25759]: osdmap e100: 8 total, 8 up, 8 in 2024-04-12T22:01:23.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:23 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/3604975852' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/903882336"}]: dispatch 2024-04-12T22:01:23.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:23 smithi114 ceph-mon[25759]: from='client.? ' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/903882336"}]: dispatch 2024-04-12T22:01:23.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:23 smithi114 ceph-mon[25759]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-12T22:01:23.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:23 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/2391241910' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/1069516662"}]': finished 2024-04-12T22:01:23.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:23 smithi097 ceph-mon[24835]: osdmap e100: 8 total, 8 up, 8 in 2024-04-12T22:01:23.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:23 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/3604975852' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/903882336"}]: dispatch 2024-04-12T22:01:23.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:23 smithi097 ceph-mon[24835]: from='client.? ' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/903882336"}]: dispatch 2024-04-12T22:01:23.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:23 smithi097 ceph-mon[24835]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-12T22:01:23.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:23 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/2391241910' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/1069516662"}]': finished 2024-04-12T22:01:23.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:23 smithi097 ceph-mon[29620]: osdmap e100: 8 total, 8 up, 8 in 2024-04-12T22:01:23.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:23 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/3604975852' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/903882336"}]: dispatch 2024-04-12T22:01:23.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:23 smithi097 ceph-mon[29620]: from='client.? ' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/903882336"}]: dispatch 2024-04-12T22:01:23.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:23 smithi097 ceph-mon[29620]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-12T22:01:24.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:24 smithi114 ceph-mon[25759]: from='client.? ' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/903882336"}]': finished 2024-04-12T22:01:24.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:24 smithi114 ceph-mon[25759]: osdmap e101: 8 total, 8 up, 8 in 2024-04-12T22:01:24.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:24 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/3994549864' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/2307085899"}]: dispatch 2024-04-12T22:01:24.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:24 smithi114 ceph-mon[25759]: from='client.? ' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/2307085899"}]: dispatch 2024-04-12T22:01:24.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:24 smithi097 ceph-mon[24835]: from='client.? ' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/903882336"}]': finished 2024-04-12T22:01:24.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:24 smithi097 ceph-mon[24835]: osdmap e101: 8 total, 8 up, 8 in 2024-04-12T22:01:24.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:24 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/3994549864' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/2307085899"}]: dispatch 2024-04-12T22:01:24.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:24 smithi097 ceph-mon[24835]: from='client.? ' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/2307085899"}]: dispatch 2024-04-12T22:01:24.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:24 smithi097 ceph-mon[29620]: from='client.? ' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/903882336"}]': finished 2024-04-12T22:01:24.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:24 smithi097 ceph-mon[29620]: osdmap e101: 8 total, 8 up, 8 in 2024-04-12T22:01:24.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:24 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/3994549864' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/2307085899"}]: dispatch 2024-04-12T22:01:24.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:24 smithi097 ceph-mon[29620]: from='client.? ' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/2307085899"}]: dispatch 2024-04-12T22:01:25.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:25 smithi114 ceph-mon[25759]: from='client.? ' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/2307085899"}]': finished 2024-04-12T22:01:25.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:25 smithi114 ceph-mon[25759]: osdmap e102: 8 total, 8 up, 8 in 2024-04-12T22:01:25.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:25 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/3661280785' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/652135493"}]: dispatch 2024-04-12T22:01:25.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:25 smithi114 ceph-mon[25759]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:01:25.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:25 smithi097 ceph-mon[24835]: from='client.? ' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/2307085899"}]': finished 2024-04-12T22:01:25.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:25 smithi097 ceph-mon[24835]: osdmap e102: 8 total, 8 up, 8 in 2024-04-12T22:01:25.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:25 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/3661280785' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/652135493"}]: dispatch 2024-04-12T22:01:25.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:25 smithi097 ceph-mon[24835]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:01:25.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:25 smithi097 ceph-mon[29620]: from='client.? ' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/2307085899"}]': finished 2024-04-12T22:01:25.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:25 smithi097 ceph-mon[29620]: osdmap e102: 8 total, 8 up, 8 in 2024-04-12T22:01:25.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:25 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/3661280785' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/652135493"}]: dispatch 2024-04-12T22:01:25.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:25 smithi097 ceph-mon[29620]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:01:26.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:26 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/3661280785' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/652135493"}]': finished 2024-04-12T22:01:26.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:26 smithi097 ceph-mon[29620]: osdmap e103: 8 total, 8 up, 8 in 2024-04-12T22:01:26.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:26 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/1055222864' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/1122129972"}]: dispatch 2024-04-12T22:01:26.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:26 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/3661280785' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/652135493"}]': finished 2024-04-12T22:01:26.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:26 smithi097 ceph-mon[24835]: osdmap e103: 8 total, 8 up, 8 in 2024-04-12T22:01:26.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:26 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/1055222864' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/1122129972"}]: dispatch 2024-04-12T22:01:26.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:26 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/3661280785' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/652135493"}]': finished 2024-04-12T22:01:26.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:26 smithi114 ceph-mon[25759]: osdmap e103: 8 total, 8 up, 8 in 2024-04-12T22:01:26.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:26 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/1055222864' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/1122129972"}]: dispatch 2024-04-12T22:01:27.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:27 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/1055222864' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/1122129972"}]': finished 2024-04-12T22:01:27.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:27 smithi097 ceph-mon[24835]: osdmap e104: 8 total, 8 up, 8 in 2024-04-12T22:01:27.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:27 smithi097 ceph-mon[24835]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:01:27.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:27 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/1055222864' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/1122129972"}]': finished 2024-04-12T22:01:27.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:27 smithi097 ceph-mon[29620]: osdmap e104: 8 total, 8 up, 8 in 2024-04-12T22:01:27.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:27 smithi097 ceph-mon[29620]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:01:27.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:27 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/1055222864' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/1122129972"}]': finished 2024-04-12T22:01:27.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:27 smithi114 ceph-mon[25759]: osdmap e104: 8 total, 8 up, 8 in 2024-04-12T22:01:27.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:27 smithi114 ceph-mon[25759]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:01:29.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:29 smithi097 ceph-mon[29620]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 914 B/s rd, 0 op/s 2024-04-12T22:01:29.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:29 smithi097 ceph-mon[29620]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:01:29.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:29 smithi097 ceph-mon[29620]: from='client.24911 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:01:29.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:29 smithi097 ceph-mon[24835]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 914 B/s rd, 0 op/s 2024-04-12T22:01:29.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:29 smithi097 ceph-mon[24835]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:01:29.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:29 smithi097 ceph-mon[24835]: from='client.24911 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:01:29.671 INFO:teuthology.orchestra.run.smithi097.stdout:true 2024-04-12T22:01:29.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:29 smithi114 ceph-mon[25759]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 914 B/s rd, 0 op/s 2024-04-12T22:01:29.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:29 smithi114 ceph-mon[25759]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:01:29.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:29 smithi114 ceph-mon[25759]: from='client.24911 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:01:30.309 INFO:teuthology.orchestra.run.smithi097.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T22:01:30.309 INFO:teuthology.orchestra.run.smithi097.stdout:alertmanager.a smithi097 *:9093,9094 starting - - - - 2024-04-12T22:01:30.310 INFO:teuthology.orchestra.run.smithi097.stdout:grafana.a smithi114 *:3000 running (5m) 51s ago 5m 52.7M - 8.3.5 dad864ee21e9 daa3c7921258 2024-04-12T22:01:30.310 INFO:teuthology.orchestra.run.smithi097.stdout:iscsi.foo.smithi097.xndkxy smithi097 running (4m) 52s ago 4m 53.6M - 3.5 e1d6a67b021e 2d79ea3caa09 2024-04-12T22:01:30.310 INFO:teuthology.orchestra.run.smithi097.stdout:mgr.x smithi114 *:8443,9283 running (72s) 51s ago 9m 494M - 19.0.0-2053-gb277feb2 668895944f0e 87e5b6de56ea 2024-04-12T22:01:30.310 INFO:teuthology.orchestra.run.smithi097.stdout:mgr.y smithi097 *:9283 running (11m) 52s ago 11m 382M - 17.2.0 e1d6a67b021e 666702929f2f 2024-04-12T22:01:30.310 INFO:teuthology.orchestra.run.smithi097.stdout:mon.a smithi097 running (11m) 52s ago 11m 53.8M 2048M 17.2.0 e1d6a67b021e 6e618120f594 2024-04-12T22:01:30.310 INFO:teuthology.orchestra.run.smithi097.stdout:mon.b smithi114 running (10m) 51s ago 10m 46.5M 2048M 17.2.0 e1d6a67b021e 47a08e70f856 2024-04-12T22:01:30.310 INFO:teuthology.orchestra.run.smithi097.stdout:mon.c smithi097 running (10m) 52s ago 10m 47.0M 2048M 17.2.0 e1d6a67b021e 679b33a29a0c 2024-04-12T22:01:30.310 INFO:teuthology.orchestra.run.smithi097.stdout:node-exporter.a smithi097 *:9100 starting - - - - 2024-04-12T22:01:30.310 INFO:teuthology.orchestra.run.smithi097.stdout:node-exporter.b smithi114 *:9100 starting - - - - 2024-04-12T22:01:30.310 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0 smithi097 running (9m) 52s ago 9m 53.1M 2736M 17.2.0 e1d6a67b021e abac1d8884fa 2024-04-12T22:01:30.310 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1 smithi097 running (9m) 52s ago 9m 52.0M 2736M 17.2.0 e1d6a67b021e 39192562d598 2024-04-12T22:01:30.310 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2 smithi097 running (8m) 52s ago 8m 49.9M 2736M 17.2.0 e1d6a67b021e 89daf8e339c3 2024-04-12T22:01:30.310 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3 smithi097 running (8m) 52s ago 8m 50.2M 2736M 17.2.0 e1d6a67b021e 4e9a0b3862d9 2024-04-12T22:01:30.311 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4 smithi114 running (7m) 51s ago 7m 51.4M 2992M 17.2.0 e1d6a67b021e fb42c0e8e575 2024-04-12T22:01:30.311 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5 smithi114 running (7m) 51s ago 7m 49.3M 2992M 17.2.0 e1d6a67b021e 3acd4a3b61e5 2024-04-12T22:01:30.311 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6 smithi114 running (7m) 51s ago 7m 48.4M 2992M 17.2.0 e1d6a67b021e 5969baf67f41 2024-04-12T22:01:30.311 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7 smithi114 running (6m) 51s ago 6m 49.6M 2992M 17.2.0 e1d6a67b021e f68340fecf2c 2024-04-12T22:01:30.311 INFO:teuthology.orchestra.run.smithi097.stdout:prometheus.a smithi114 *:9095 starting - - - - 2024-04-12T22:01:30.311 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.foo.smithi097.sggtkh smithi097 *:8000 running (4m) 52s ago 4m 70.4M - 17.2.0 e1d6a67b021e a39f9b2d8509 2024-04-12T22:01:30.311 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.foo.smithi114.ulfbyo smithi114 *:8000 running (4m) 51s ago 4m 70.9M - 17.2.0 e1d6a67b021e df79a48bdd59 2024-04-12T22:01:30.311 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.smpl.smithi097.exxkgl smithi097 *:80 running (4m) 52s ago 4m 70.5M - 17.2.0 e1d6a67b021e a036c58a0099 2024-04-12T22:01:30.311 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.smpl.smithi114.lrgvph smithi114 *:80 running (4m) 51s ago 4m 72.4M - 17.2.0 e1d6a67b021e 031a5fc24237 2024-04-12T22:01:30.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:30 smithi097 ceph-mon[24835]: from='client.15306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:01:30.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:30 smithi097 ceph-mon[24835]: from='client.25204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:01:30.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:30 smithi097 ceph-mon[24835]: from='client.25210 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:01:30.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:30 smithi097 ceph-mon[29620]: from='client.15306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:01:30.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:30 smithi097 ceph-mon[29620]: from='client.25204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:01:30.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:30 smithi097 ceph-mon[29620]: from='client.25210 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:01:30.698 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-12T22:01:30.698 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": { 2024-04-12T22:01:30.698 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-04-12T22:01:30.698 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:01:30.698 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": { 2024-04-12T22:01:30.698 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-04-12T22:01:30.698 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 1 2024-04-12T22:01:30.699 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:01:30.699 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": { 2024-04-12T22:01:30.699 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-12T22:01:30.699 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:01:30.699 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": {}, 2024-04-12T22:01:30.699 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw": { 2024-04-12T22:01:30.699 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-12T22:01:30.699 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:01:30.699 INFO:teuthology.orchestra.run.smithi097.stdout: "overall": { 2024-04-12T22:01:30.699 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 16, 2024-04-12T22:01:30.699 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 1 2024-04-12T22:01:30.699 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-12T22:01:30.699 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-12T22:01:30.700 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:30 smithi114 ceph-mon[25759]: from='client.15306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:01:30.700 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:30 smithi114 ceph-mon[25759]: from='client.25204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:01:30.700 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:30 smithi114 ceph-mon[25759]: from='client.25210 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:01:31.027 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-12T22:01:31.028 INFO:teuthology.orchestra.run.smithi097.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a", 2024-04-12T22:01:31.028 INFO:teuthology.orchestra.run.smithi097.stdout: "in_progress": true, 2024-04-12T22:01:31.028 INFO:teuthology.orchestra.run.smithi097.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T22:01:31.028 INFO:teuthology.orchestra.run.smithi097.stdout: "services_complete": [], 2024-04-12T22:01:31.028 INFO:teuthology.orchestra.run.smithi097.stdout: "progress": "1/23 daemons upgraded", 2024-04-12T22:01:31.028 INFO:teuthology.orchestra.run.smithi097.stdout: "message": "", 2024-04-12T22:01:31.028 INFO:teuthology.orchestra.run.smithi097.stdout: "is_paused": false 2024-04-12T22:01:31.028 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-12T22:01:31.409 INFO:teuthology.orchestra.run.smithi097.stdout:HEALTH_OK 2024-04-12T22:01:31.606 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:31 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/1862882025' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:01:31.606 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:31 smithi114 ceph-mon[25759]: from='client.15330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:01:31.606 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:31 smithi114 ceph-mon[25759]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-12T22:01:31.606 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:31 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/2504499463' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T22:01:31.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:31 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/1862882025' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:01:31.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:31 smithi097 ceph-mon[29620]: from='client.15330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:01:31.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:31 smithi097 ceph-mon[29620]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-12T22:01:31.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:31 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/2504499463' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T22:01:31.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:31 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/1862882025' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:01:31.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:31 smithi097 ceph-mon[24835]: from='client.15330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:01:31.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:31 smithi097 ceph-mon[24835]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-12T22:01:31.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:31 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/2504499463' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T22:01:31.994 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:01:31 smithi114 systemd[1]: Stopping Ceph grafana.a for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:01:31.994 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:01:31 smithi114 bash[51076]: Error: no container with name or ID "ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana.a" found: no such container 2024-04-12T22:01:31.994 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:01:31 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[41845]: t=2024-04-12T22:01:31+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-04-12T22:01:31.994 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:01:31 smithi114 podman[51087]: 2024-04-12 22:01:31.753459957 +0000 UTC m=+0.113943794 container died daa3c792125863c0eee3c7c3c14b153a5626c3648a39cfecf09f2d3ef1900481 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, version=8.5, io.k8s.display-name=Red Hat Universal Base Image 8, io.buildah.version=1.24.2, summary=Grafana Container configured for Ceph mgr/dashboard integration, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=ubi8-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, description=Ceph Grafana Container, io.openshift.expose-services=, vcs-type=git, release=236.1648460182, distribution-scope=public, build-date=2022-03-28T10:36:18.413762, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.tags=base rhel8, name=ubi8) 2024-04-12T22:01:32.410 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:01:32 smithi114 podman[51087]: 2024-04-12 22:01:32.100891656 +0000 UTC m=+0.461375491 container cleanup daa3c792125863c0eee3c7c3c14b153a5626c3648a39cfecf09f2d3ef1900481 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, io.openshift.tags=base rhel8, vcs-type=git, vendor=Red Hat, Inc., description=Ceph Grafana Container, maintainer=Paul Cuzner , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.buildah.version=1.24.2, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.component=ubi8-container, summary=Grafana Container configured for Ceph mgr/dashboard integration, release=236.1648460182, architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, build-date=2022-03-28T10:36:18.413762, version=8.5) 2024-04-12T22:01:32.410 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:01:32 smithi114 bash[51087]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a 2024-04-12T22:01:32.411 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:01:32 smithi114 bash[51114]: Error: no container with name or ID "ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana.a" found: no such container 2024-04-12T22:01:32.743 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:01:32 smithi114 podman[51098]: 2024-04-12 22:01:32.41028926 +0000 UTC m=+0.653062409 container remove daa3c792125863c0eee3c7c3c14b153a5626c3648a39cfecf09f2d3ef1900481 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-03-28T10:36:18.413762, release=236.1648460182, version=8.5, distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, maintainer=Paul Cuzner , vendor=Red Hat, Inc., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.24.2, com.redhat.component=ubi8-container, summary=Grafana Container configured for Ceph mgr/dashboard integration, description=Ceph Grafana Container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-type=git) 2024-04-12T22:01:32.744 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:01:32 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@grafana.a.service: Deactivated successfully. 2024-04-12T22:01:32.744 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:01:32 smithi114 systemd[1]: Stopped Ceph grafana.a for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:01:32.744 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:01:32 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@grafana.a.service: Consumed 2.488s CPU time. 2024-04-12T22:01:32.744 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:01:32 smithi114 systemd[1]: Starting Ceph grafana.a for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:01:32.744 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:01:32 smithi114 podman[51179]: 2024-04-12 22:01:32.539583592 +0000 UTC m=+0.016721335 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-04-12T22:01:32.744 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:01:32 smithi114 podman[51179]: 2024-04-12 22:01:32.644562307 +0000 UTC m=+0.121700080 container create c643fc2ccc37ce75d5e6c14cb4b8f306fd766674e088106369896c20572e0862 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, distribution-scope=public, io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-type=git, architecture=x86_64, com.redhat.component=ubi8-container, io.buildah.version=1.24.2, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, maintainer=Paul Cuzner , description=Ceph Grafana Container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, version=8.5, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-03-28T10:36:18.413762, io.openshift.tags=base rhel8, release=236.1648460182, name=ubi8) 2024-04-12T22:01:33.019 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:33 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: [12/Apr/2024:22:01:33] ENGINE Bus STOPPING 2024-04-12T22:01:33.020 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:01:32 smithi114 podman[51179]: 2024-04-12 22:01:32.87214623 +0000 UTC m=+0.349283959 container init c643fc2ccc37ce75d5e6c14cb4b8f306fd766674e088106369896c20572e0862 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container, version=8.5, release=236.1648460182, vendor=Red Hat, Inc., io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.component=ubi8-container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.24.2, build-date=2022-03-28T10:36:18.413762, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, maintainer=Paul Cuzner , name=ubi8, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.tags=base rhel8, architecture=x86_64) 2024-04-12T22:01:33.020 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:01:32 smithi114 podman[51179]: 2024-04-12 22:01:32.874619097 +0000 UTC m=+0.351756825 container start c643fc2ccc37ce75d5e6c14cb4b8f306fd766674e088106369896c20572e0862 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, io.buildah.version=1.24.2, architecture=x86_64, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.component=ubi8-container, io.openshift.tags=base rhel8, release=236.1648460182, name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, description=Ceph Grafana Container, build-date=2022-03-28T10:36:18.413762, vendor=Red Hat, Inc., distribution-scope=public, maintainer=Paul Cuzner , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-type=git) 2024-04-12T22:01:33.020 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:01:32 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[51193]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-12T22:01:33.020 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:01:32 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[51193]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-12T22:01:33.020 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:01:32 smithi114 bash[51179]: c643fc2ccc37ce75d5e6c14cb4b8f306fd766674e088106369896c20572e0862 2024-04-12T22:01:33.020 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:01:32 smithi114 systemd[1]: Started Ceph grafana.a for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:01:33.020 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:01:32 smithi114 podman[51209]: 2024-04-12 22:01:32.947750452 +0000 UTC m=+0.039243052 container died c643fc2ccc37ce75d5e6c14cb4b8f306fd766674e088106369896c20572e0862 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, vcs-type=git, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.24.2, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, distribution-scope=public, com.redhat.component=ubi8-container, description=Ceph Grafana Container, io.openshift.expose-services=, name=ubi8, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=236.1648460182, maintainer=Paul Cuzner , summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base rhel8, vendor=Red Hat, Inc., build-date=2022-03-28T10:36:18.413762, version=8.5) 2024-04-12T22:01:33.276 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:33 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: [12/Apr/2024:22:01:33] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-12T22:01:33.277 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:33 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: [12/Apr/2024:22:01:33] ENGINE Bus STOPPED 2024-04-12T22:01:33.277 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:33 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: [12/Apr/2024:22:01:33] ENGINE Bus STARTING 2024-04-12T22:01:33.277 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:33 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: [12/Apr/2024:22:01:33] ENGINE Serving on http://:::9283 2024-04-12T22:01:33.277 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:33 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: [12/Apr/2024:22:01:33] ENGINE Bus STARTED 2024-04-12T22:01:33.277 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:33 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: [12/Apr/2024:22:01:33] ENGINE Bus STOPPING 2024-04-12T22:01:33.277 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:33 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: [12/Apr/2024:22:01:33] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-12T22:01:33.277 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:33 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: [12/Apr/2024:22:01:33] ENGINE Bus STOPPED 2024-04-12T22:01:33.277 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:33 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: [12/Apr/2024:22:01:33] ENGINE Bus STARTING 2024-04-12T22:01:33.277 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:33 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: [12/Apr/2024:22:01:33] ENGINE Serving on http://:::9283 2024-04-12T22:01:33.277 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:33 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: [12/Apr/2024:22:01:33] ENGINE Bus STARTED 2024-04-12T22:01:33.277 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:33 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: [12/Apr/2024:22:01:33] ENGINE Bus STOPPING 2024-04-12T22:01:33.277 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:33 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: [12/Apr/2024:22:01:33] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-12T22:01:33.277 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:33 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: [12/Apr/2024:22:01:33] ENGINE Bus STOPPED 2024-04-12T22:01:33.278 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:33 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: [12/Apr/2024:22:01:33] ENGINE Bus STARTING 2024-04-12T22:01:33.580 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:33 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: [12/Apr/2024:22:01:33] ENGINE Serving on http://:::9283 2024-04-12T22:01:33.580 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:33 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: [12/Apr/2024:22:01:33] ENGINE Bus STARTED 2024-04-12T22:01:33.974 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:33 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:01:33] "GET /metrics HTTP/1.1" 200 38153 "" "Prometheus/2.33.4" 2024-04-12T22:01:33.974 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:01:33 smithi114 podman[51209]: 2024-04-12 22:01:33.592539853 +0000 UTC m=+0.684032432 container remove c643fc2ccc37ce75d5e6c14cb4b8f306fd766674e088106369896c20572e0862 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, build-date=2022-03-28T10:36:18.413762, description=Ceph Grafana Container, maintainer=Paul Cuzner , name=ubi8, vendor=Red Hat, Inc., io.openshift.tags=base rhel8, io.openshift.expose-services=, io.buildah.version=1.24.2, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=236.1648460182, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI) 2024-04-12T22:01:33.974 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:01:33 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-04-12T22:01:33.974 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:01:33 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-04-12T22:01:34.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:33 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:34.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:33 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:34.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:33 smithi114 ceph-mon[25759]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-12T22:01:34.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:33 smithi114 ceph-mon[25759]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-12T22:01:34.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:33 smithi114 ceph-mon[25759]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi097.front.sepia.ceph.com:9093"}]: dispatch 2024-04-12T22:01:34.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:33 smithi114 ceph-mon[25759]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi097.front.sepia.ceph.com:9093"}]: dispatch 2024-04-12T22:01:34.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:33 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:34.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:33 smithi114 ceph-mon[25759]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-12T22:01:34.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:33 smithi114 ceph-mon[25759]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-12T22:01:34.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:33 smithi114 ceph-mon[25759]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi114.front.sepia.ceph.com:9095"}]: dispatch 2024-04-12T22:01:34.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:33 smithi114 ceph-mon[25759]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi114.front.sepia.ceph.com:9095"}]: dispatch 2024-04-12T22:01:34.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:33 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:34.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:33 smithi114 ceph-mon[25759]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-12T22:01:34.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:33 smithi114 ceph-mon[25759]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-12T22:01:34.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:33 smithi114 ceph-mon[25759]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi114.front.sepia.ceph.com:3000"}]: dispatch 2024-04-12T22:01:34.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:33 smithi114 ceph-mon[25759]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi114.front.sepia.ceph.com:3000"}]: dispatch 2024-04-12T22:01:34.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:33 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:34.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:33 smithi114 ceph-mon[25759]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:01:34.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:33 smithi114 ceph-mon[25759]: Upgrade: Need to upgrade myself (mgr.x) 2024-04-12T22:01:34.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:33 smithi114 ceph-mon[25759]: Upgrade: Need to upgrade myself (mgr.x) 2024-04-12T22:01:34.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:33 smithi114 ceph-mon[25759]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-12T22:01:34.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:33 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:34.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:33 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:34.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:33 smithi097 ceph-mon[29620]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-12T22:01:34.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:33 smithi097 ceph-mon[29620]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-12T22:01:34.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:33 smithi097 ceph-mon[29620]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi097.front.sepia.ceph.com:9093"}]: dispatch 2024-04-12T22:01:34.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:33 smithi097 ceph-mon[29620]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi097.front.sepia.ceph.com:9093"}]: dispatch 2024-04-12T22:01:34.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:33 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:34.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:33 smithi097 ceph-mon[29620]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-12T22:01:34.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:33 smithi097 ceph-mon[29620]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-12T22:01:34.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:33 smithi097 ceph-mon[29620]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi114.front.sepia.ceph.com:9095"}]: dispatch 2024-04-12T22:01:34.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:33 smithi097 ceph-mon[29620]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi114.front.sepia.ceph.com:9095"}]: dispatch 2024-04-12T22:01:34.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:33 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:34.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:33 smithi097 ceph-mon[29620]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-12T22:01:34.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:33 smithi097 ceph-mon[29620]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-12T22:01:34.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:33 smithi097 ceph-mon[29620]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi114.front.sepia.ceph.com:3000"}]: dispatch 2024-04-12T22:01:34.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:33 smithi097 ceph-mon[29620]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi114.front.sepia.ceph.com:3000"}]: dispatch 2024-04-12T22:01:34.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:33 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:34.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:33 smithi097 ceph-mon[29620]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:01:34.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:33 smithi097 ceph-mon[29620]: Upgrade: Need to upgrade myself (mgr.x) 2024-04-12T22:01:34.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:33 smithi097 ceph-mon[29620]: Upgrade: Need to upgrade myself (mgr.x) 2024-04-12T22:01:34.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:33 smithi097 ceph-mon[29620]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-12T22:01:34.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:33 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:34.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:33 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:34.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:33 smithi097 ceph-mon[24835]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-12T22:01:34.413 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:33 smithi097 ceph-mon[24835]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-12T22:01:34.413 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:33 smithi097 ceph-mon[24835]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi097.front.sepia.ceph.com:9093"}]: dispatch 2024-04-12T22:01:34.413 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:33 smithi097 ceph-mon[24835]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi097.front.sepia.ceph.com:9093"}]: dispatch 2024-04-12T22:01:34.413 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:33 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:34.413 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:33 smithi097 ceph-mon[24835]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-12T22:01:34.413 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:33 smithi097 ceph-mon[24835]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-12T22:01:34.413 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:33 smithi097 ceph-mon[24835]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi114.front.sepia.ceph.com:9095"}]: dispatch 2024-04-12T22:01:34.413 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:33 smithi097 ceph-mon[24835]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi114.front.sepia.ceph.com:9095"}]: dispatch 2024-04-12T22:01:34.413 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:33 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:34.413 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:33 smithi097 ceph-mon[24835]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-12T22:01:34.413 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:33 smithi097 ceph-mon[24835]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-12T22:01:34.413 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:33 smithi097 ceph-mon[24835]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi114.front.sepia.ceph.com:3000"}]: dispatch 2024-04-12T22:01:34.413 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:33 smithi097 ceph-mon[24835]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi114.front.sepia.ceph.com:3000"}]: dispatch 2024-04-12T22:01:34.413 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:33 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:34.414 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:33 smithi097 ceph-mon[24835]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:01:34.414 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:33 smithi097 ceph-mon[24835]: Upgrade: Need to upgrade myself (mgr.x) 2024-04-12T22:01:34.414 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:33 smithi097 ceph-mon[24835]: Upgrade: Need to upgrade myself (mgr.x) 2024-04-12T22:01:34.414 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:33 smithi097 ceph-mon[24835]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-12T22:01:35.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:35 smithi097 ceph-mon[24835]: Upgrade: Updating mgr.y 2024-04-12T22:01:35.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:35 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:35.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:35 smithi097 ceph-mon[24835]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T22:01:35.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:35 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T22:01:35.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:35 smithi097 ceph-mon[24835]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T22:01:35.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:35 smithi097 ceph-mon[24835]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:01:35.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:35 smithi097 ceph-mon[24835]: Deploying daemon mgr.y on smithi097 2024-04-12T22:01:35.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:35 smithi097 ceph-mon[24835]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-12T22:01:35.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:35 smithi097 ceph-mon[29620]: Upgrade: Updating mgr.y 2024-04-12T22:01:35.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:35 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:35.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:35 smithi097 ceph-mon[29620]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T22:01:35.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:35 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T22:01:35.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:35 smithi097 ceph-mon[29620]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T22:01:35.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:35 smithi097 ceph-mon[29620]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:01:35.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:35 smithi097 ceph-mon[29620]: Deploying daemon mgr.y on smithi097 2024-04-12T22:01:35.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:35 smithi097 ceph-mon[29620]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-12T22:01:35.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:35 smithi114 ceph-mon[25759]: Upgrade: Updating mgr.y 2024-04-12T22:01:35.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:35 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:35.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:35 smithi114 ceph-mon[25759]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T22:01:35.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:35 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T22:01:35.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:35 smithi114 ceph-mon[25759]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T22:01:35.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:35 smithi114 ceph-mon[25759]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:01:35.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:35 smithi114 ceph-mon[25759]: Deploying daemon mgr.y on smithi097 2024-04-12T22:01:35.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:35 smithi114 ceph-mon[25759]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-12T22:01:36.868 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:01:36 smithi097 systemd[1]: Stopping Ceph mgr.y for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:01:37.137 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:01:36 smithi097 podman[60395]: 2024-04-12 22:01:36.866767018 +0000 UTC m=+0.135318768 container died 666702929f2fe155531b14883cbdcc9019584f736e3cccd98adb23c278925ec5 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, ceph=True, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, vcs-type=git, distribution-scope=public, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, architecture=x86_64, vendor=Red Hat, Inc.) 2024-04-12T22:01:37.410 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:01:37 smithi097 podman[60395]: 2024-04-12 22:01:37.202085643 +0000 UTC m=+0.470637384 container cleanup 666702929f2fe155531b14883cbdcc9019584f736e3cccd98adb23c278925ec5 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , version=8, io.buildah.version=1.19.8, io.openshift.expose-services=, RELEASE=HEAD, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, ceph=True, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-04-12T22:01:37.410 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:01:37 smithi097 bash[60395]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y 2024-04-12T22:01:37.411 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:01:37 smithi097 podman[60409]: 2024-04-12 22:01:37.378532577 +0000 UTC m=+0.516733322 container remove 666702929f2fe155531b14883cbdcc9019584f736e3cccd98adb23c278925ec5 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, distribution-scope=public, release=754, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, ceph=True, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, vcs-type=git, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0) 2024-04-12T22:01:37.411 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:01:37 smithi097 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-04-12T22:01:37.910 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:01:37 smithi097 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@mgr.y.service: Failed with result 'exit-code'. 2024-04-12T22:01:37.910 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:01:37 smithi097 systemd[1]: Stopped Ceph mgr.y for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:01:37.911 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:01:37 smithi097 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@mgr.y.service: Consumed 53.267s CPU time. 2024-04-12T22:01:37.911 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:01:37 smithi097 systemd[1]: Starting Ceph mgr.y for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:01:38.293 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:38 smithi097 ceph-mon[24835]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 967 B/s rd, 0 op/s 2024-04-12T22:01:38.294 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:38 smithi097 ceph-mon[29620]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 967 B/s rd, 0 op/s 2024-04-12T22:01:38.294 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:01:38 smithi097 podman[60540]: 2024-04-12 22:01:37.931842252 +0000 UTC m=+0.017154927 image pull 668895944f0ec64b67a5b7564961d2756f2966e5d4b9338516300dbec9984951 quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a 2024-04-12T22:01:38.294 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:01:38 smithi097 podman[60540]: 2024-04-12 22:01:38.046563396 +0000 UTC m=+0.131876072 container create 488170dac11fe698b3992142aa635667fb319f2377e61c1ec5d628c645d20daa (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-b277feb) 2024-04-12T22:01:38.294 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:01:38 smithi097 podman[60540]: 2024-04-12 22:01:38.266041099 +0000 UTC m=+0.351353783 container init 488170dac11fe698b3992142aa635667fb319f2377e61c1ec5d628c645d20daa (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240409, RELEASE=squid-b277feb, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-12T22:01:38.294 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:01:38 smithi097 podman[60540]: 2024-04-12 22:01:38.268789725 +0000 UTC m=+0.354102402 container start 488170dac11fe698b3992142aa635667fb319f2377e61c1ec5d628c645d20daa (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-b277feb, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-12T22:01:38.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:38 smithi114 ceph-mon[25759]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 967 B/s rd, 0 op/s 2024-04-12T22:01:38.660 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:01:38 smithi097 ceph-mgr[60559]: -- 172.21.15.97:0/1046725443 <== mon.0 v2:172.21.15.97:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558812dc05a0 con 0x558812d94c00 2024-04-12T22:01:38.660 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:01:38 smithi097 bash[60540]: 488170dac11fe698b3992142aa635667fb319f2377e61c1ec5d628c645d20daa 2024-04-12T22:01:38.660 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:01:38 smithi097 systemd[1]: Started Ceph mgr.y for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:01:38.993 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:01:38 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[50546]: ts=2024-04-12T22:01:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"smithi114:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"172.21.15.114:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-12T22:01:39.160 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:01:38 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:01:38.835+0000 7f63b6cd4200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-12T22:01:39.160 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:01:38 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:01:38.934+0000 7f63b6cd4200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-12T22:01:39.648 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:39 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:39.648 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:39 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:39.648 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:39 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:39.648 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:39 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:39.648 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:39 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:39.648 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:39 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:39.648 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:39 smithi097 ceph-mon[24835]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:01:39.648 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:39 smithi097 ceph-mon[24835]: from='client.24911 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:01:39.649 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:39 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:39.649 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:39 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:39.649 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:39 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:39.649 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:39 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:39.649 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:39 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:39.649 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:39 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:39.649 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:39 smithi097 ceph-mon[29620]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:01:39.649 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:39 smithi097 ceph-mon[29620]: from='client.24911 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:01:39.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:39 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:39.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:39 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:39.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:39 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:39.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:39 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:39.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:39 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:39.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:39 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:39.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:39 smithi114 ceph-mon[25759]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:01:39.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:39 smithi114 ceph-mon[25759]: from='client.24911 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:01:39.905 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:01:39 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:01:39.646+0000 7f63b6cd4200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-12T22:01:39.906 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:01:39 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:01:39.755+0000 7f63b6cd4200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-12T22:01:40.160 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:01:39 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:01:39.904+0000 7f63b6cd4200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-12T22:01:40.160 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:01:40 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:01:40.008+0000 7f63b6cd4200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-12T22:01:40.160 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:01:40 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:01:40.096+0000 7f63b6cd4200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-12T22:01:40.494 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:01:40 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:01:40.382+0000 7f63b6cd4200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-12T22:01:40.495 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:01:40 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:01:40.493+0000 7f63b6cd4200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-12T22:01:40.841 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:01:40 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:01:40.585+0000 7f63b6cd4200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-12T22:01:40.841 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:01:40 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:01:40.689+0000 7f63b6cd4200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-12T22:01:41.160 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:01:40 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:01:40.872+0000 7f63b6cd4200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-12T22:01:42.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:42 smithi097 ceph-mon[24835]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:01:42.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:42 smithi097 ceph-mon[29620]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:01:42.161 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:01:41 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:01:41.901+0000 7f63b6cd4200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-12T22:01:42.161 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:01:42 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:01:42.043+0000 7f63b6cd4200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-12T22:01:42.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:42 smithi114 ceph-mon[25759]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:01:42.660 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:01:42 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:01:42.409+0000 7f63b6cd4200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-12T22:01:42.660 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:01:42 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:01:42.503+0000 7f63b6cd4200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-12T22:01:43.160 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:01:42 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:01:42.706+0000 7f63b6cd4200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-12T22:01:43.160 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:01:42 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:01:42.910+0000 7f63b6cd4200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-12T22:01:43.490 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:01:43 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:01:43.218+0000 7f63b6cd4200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-12T22:01:43.490 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:01:43 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:01:43.489+0000 7f63b6cd4200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-12T22:01:43.910 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:01:43 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:01:43.734+0000 7f63b6cd4200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-12T22:01:43.911 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:01:43 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[59315]: level=warn ts=2024-04-12T22:01:43.508Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi097.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.97:8443: connect: connection refused" 2024-04-12T22:01:43.994 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:43 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:01:43] "GET /metrics HTTP/1.1" 200 38153 "" "Prometheus/2.33.4" 2024-04-12T22:01:43.994 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:01:43 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@grafana.a.service: Scheduled restart job, restart counter is at 1. 2024-04-12T22:01:43.994 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:01:43 smithi114 systemd[1]: Stopped Ceph grafana.a for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:01:43.994 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:01:43 smithi114 systemd[1]: Starting Ceph grafana.a for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:01:44.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:44 smithi097 ceph-mon[24835]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:01:44.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:44 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:44.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:44 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:44.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:44 smithi097 ceph-mon[29620]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:01:44.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:44 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:44.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:44 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:44.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:44 smithi114 ceph-mon[25759]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:01:44.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:44 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:44.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:44 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:44.910 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:01:44 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:01:44.597+0000 7f63b6cd4200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-12T22:01:45.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:45 smithi097 ceph-mon[24835]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:01:45.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:45 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:45.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:45 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:45.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:45 smithi097 ceph-mon[29620]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:01:45.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:45 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:45.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:45 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:45.411 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:01:45 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:01:45.202+0000 7f63b6cd4200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-12T22:01:45.411 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:01:45 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:01:45.307+0000 7f63b6cd4200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-12T22:01:45.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:45 smithi114 ceph-mon[25759]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:01:45.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:45 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:45.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:45 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:45.910 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:01:45 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:01:45.546+0000 7f63b6cd4200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-12T22:01:45.911 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:01:45 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: [12/Apr/2024:22:01:45] ENGINE Bus STARTING 2024-04-12T22:01:45.911 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:01:45 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: CherryPy Checker: 2024-04-12T22:01:45.911 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:01:45 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: The Application mounted at '' has an empty config. 2024-04-12T22:01:45.911 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:01:45 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:01:45.911 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:01:45 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: [12/Apr/2024:22:01:45] ENGINE Serving on http://:::9283 2024-04-12T22:01:45.911 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:01:45 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: [12/Apr/2024:22:01:45] ENGINE Bus STARTED 2024-04-12T22:01:45.993 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:01:45 smithi114 podman[51648]: 2024-04-12 22:01:45.549335531 +0000 UTC m=+0.020697630 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-04-12T22:01:45.993 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:01:45 smithi114 podman[51648]: 2024-04-12 22:01:45.71660255 +0000 UTC m=+0.187964646 container create 5ef2c308060116b186861b0d1e9268ffb60d6f7f4aae3a475560aceec4235561 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, io.buildah.version=1.24.2, release=236.1648460182, distribution-scope=public, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, build-date=2022-03-28T10:36:18.413762, io.openshift.expose-services=, description=Ceph Grafana Container, com.redhat.component=ubi8-container, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.tags=base rhel8, maintainer=Paul Cuzner , com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64) 2024-04-12T22:01:46.410 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:01:45 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[59315]: level=info ts=2024-04-12T22:01:45.983Z caller=notify.go:735 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify success" attempts=4 2024-04-12T22:01:46.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:46 smithi097 ceph-mon[24835]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:01:46.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:46 smithi097 ceph-mon[24835]: Standby manager daemon y restarted 2024-04-12T22:01:46.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:46 smithi097 ceph-mon[24835]: Standby manager daemon y started 2024-04-12T22:01:46.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:46 smithi097 ceph-mon[24835]: from='mgr.? 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-12T22:01:46.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:46 smithi097 ceph-mon[24835]: from='mgr.? 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-12T22:01:46.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:46 smithi097 ceph-mon[24835]: from='mgr.? 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-12T22:01:46.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:46 smithi097 ceph-mon[24835]: from='mgr.? 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-12T22:01:46.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:46 smithi097 ceph-mon[29620]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:01:46.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:46 smithi097 ceph-mon[29620]: Standby manager daemon y restarted 2024-04-12T22:01:46.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:46 smithi097 ceph-mon[29620]: Standby manager daemon y started 2024-04-12T22:01:46.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:46 smithi097 ceph-mon[29620]: from='mgr.? 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-12T22:01:46.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:46 smithi097 ceph-mon[29620]: from='mgr.? 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-12T22:01:46.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:46 smithi097 ceph-mon[29620]: from='mgr.? 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-12T22:01:46.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:46 smithi097 ceph-mon[29620]: from='mgr.? 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-12T22:01:46.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:46 smithi114 ceph-mon[25759]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:01:46.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:46 smithi114 ceph-mon[25759]: Standby manager daemon y restarted 2024-04-12T22:01:46.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:46 smithi114 ceph-mon[25759]: Standby manager daemon y started 2024-04-12T22:01:46.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:46 smithi114 ceph-mon[25759]: from='mgr.? 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-12T22:01:46.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:46 smithi114 ceph-mon[25759]: from='mgr.? 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-12T22:01:46.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:46 smithi114 ceph-mon[25759]: from='mgr.? 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-12T22:01:46.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:46 smithi114 ceph-mon[25759]: from='mgr.? 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-12T22:01:46.494 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:01:46 smithi114 podman[51648]: 2024-04-12 22:01:46.126328015 +0000 UTC m=+0.597690103 container init 5ef2c308060116b186861b0d1e9268ffb60d6f7f4aae3a475560aceec4235561 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-type=git, maintainer=Paul Cuzner , build-date=2022-03-28T10:36:18.413762, version=8.5, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=ubi8-container, release=236.1648460182, io.openshift.tags=base rhel8, distribution-scope=public, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, name=ubi8, io.openshift.expose-services=, description=Ceph Grafana Container, io.buildah.version=1.24.2) 2024-04-12T22:01:46.494 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:01:46 smithi114 podman[51648]: 2024-04-12 22:01:46.129232464 +0000 UTC m=+0.600594553 container start 5ef2c308060116b186861b0d1e9268ffb60d6f7f4aae3a475560aceec4235561 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, version=8.5, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, io.k8s.display-name=Red Hat Universal Base Image 8, vendor=Red Hat, Inc., maintainer=Paul Cuzner , distribution-scope=public, summary=Grafana Container configured for Ceph mgr/dashboard integration, release=236.1648460182, architecture=x86_64, description=Ceph Grafana Container, io.buildah.version=1.24.2, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-03-28T10:36:18.413762, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.tags=base rhel8, name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.component=ubi8-container) 2024-04-12T22:01:46.494 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:01:46 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[51682]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-12T22:01:46.495 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:01:46 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[51682]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-12T22:01:46.495 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:01:46 smithi114 bash[51648]: 5ef2c308060116b186861b0d1e9268ffb60d6f7f4aae3a475560aceec4235561 2024-04-12T22:01:46.495 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:01:46 smithi114 systemd[1]: Started Ceph grafana.a for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:01:46.495 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:01:46 smithi114 podman[51717]: 2024-04-12 22:01:46.240856274 +0000 UTC m=+0.073345361 container died 5ef2c308060116b186861b0d1e9268ffb60d6f7f4aae3a475560aceec4235561 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, io.openshift.tags=base rhel8, maintainer=Paul Cuzner , vendor=Red Hat, Inc., architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=ubi8-container, vcs-type=git, version=8.5, io.buildah.version=1.24.2, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, distribution-scope=public, release=236.1648460182, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, description=Ceph Grafana Container, name=ubi8, build-date=2022-03-28T10:36:18.413762, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI) 2024-04-12T22:01:46.993 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:01:46 smithi114 podman[51717]: 2024-04-12 22:01:46.892011386 +0000 UTC m=+0.724500508 container remove 5ef2c308060116b186861b0d1e9268ffb60d6f7f4aae3a475560aceec4235561 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, io.openshift.expose-services=, name=ubi8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, vcs-type=git, description=Ceph Grafana Container, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.tags=base rhel8, build-date=2022-03-28T10:36:18.413762, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=ubi8-container, version=8.5, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vendor=Red Hat, Inc., distribution-scope=public, io.buildah.version=1.24.2, architecture=x86_64, release=236.1648460182) 2024-04-12T22:01:46.993 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:01:46 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-04-12T22:01:46.993 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:01:46 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-04-12T22:01:47.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:47 smithi097 ceph-mon[24835]: mgrmap e26: x(active, since 78s), standbys: y 2024-04-12T22:01:47.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:47 smithi097 ceph-mon[29620]: mgrmap e26: x(active, since 78s), standbys: y 2024-04-12T22:01:47.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:47 smithi114 ceph-mon[25759]: mgrmap e26: x(active, since 78s), standbys: y 2024-04-12T22:01:48.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:48 smithi114 ceph-mon[25759]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:01:48.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:48 smithi097 ceph-mon[24835]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:01:48.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:48 smithi097 ceph-mon[29620]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:01:48.993 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:01:48 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[50546]: ts=2024-04-12T22:01:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"smithi114:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"172.21.15.114:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-12T22:01:49.910 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:49 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:49.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:49 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:49.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:49 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:49.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:49 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:49.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:49 smithi097 ceph-mon[29620]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:01:49.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:49 smithi097 ceph-mon[29620]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:01:49.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:49 smithi097 ceph-mon[29620]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:01:49.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:49 smithi097 ceph-mon[29620]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1001 B/s rd, 0 op/s 2024-04-12T22:01:49.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:49 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:49.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:49 smithi097 ceph-mon[29620]: from='client.24911 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:01:49.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:49 smithi097 ceph-mon[29620]: Reconfiguring iscsi.foo.smithi097.xndkxy (dependencies changed)... 2024-04-12T22:01:49.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:49 smithi097 ceph-mon[29620]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi097.xndkxy", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-12T22:01:49.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:49 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi097.xndkxy", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-12T22:01:49.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:49 smithi097 ceph-mon[29620]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:01:49.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:49 smithi097 ceph-mon[29620]: Reconfiguring daemon iscsi.foo.smithi097.xndkxy on smithi097 2024-04-12T22:01:49.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:49 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:49.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:49 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:49.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:49 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:49.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:49 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:49.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:49 smithi097 ceph-mon[24835]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:01:49.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:49 smithi097 ceph-mon[24835]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:01:49.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:49 smithi097 ceph-mon[24835]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:01:49.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:49 smithi097 ceph-mon[24835]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1001 B/s rd, 0 op/s 2024-04-12T22:01:49.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:49 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:49.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:49 smithi097 ceph-mon[24835]: from='client.24911 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:01:49.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:49 smithi097 ceph-mon[24835]: Reconfiguring iscsi.foo.smithi097.xndkxy (dependencies changed)... 2024-04-12T22:01:49.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:49 smithi097 ceph-mon[24835]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi097.xndkxy", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-12T22:01:49.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:49 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi097.xndkxy", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-12T22:01:49.913 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:49 smithi097 ceph-mon[24835]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:01:49.913 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:49 smithi097 ceph-mon[24835]: Reconfiguring daemon iscsi.foo.smithi097.xndkxy on smithi097 2024-04-12T22:01:49.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:49 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:49.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:49 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:49.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:49 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:49.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:49 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:49.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:49 smithi114 ceph-mon[25759]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:01:49.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:49 smithi114 ceph-mon[25759]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:01:49.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:49 smithi114 ceph-mon[25759]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:01:49.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:49 smithi114 ceph-mon[25759]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1001 B/s rd, 0 op/s 2024-04-12T22:01:49.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:49 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' 2024-04-12T22:01:49.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:49 smithi114 ceph-mon[25759]: from='client.24911 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:01:49.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:49 smithi114 ceph-mon[25759]: Reconfiguring iscsi.foo.smithi097.xndkxy (dependencies changed)... 2024-04-12T22:01:49.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:49 smithi114 ceph-mon[25759]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi097.xndkxy", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-12T22:01:49.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:49 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi097.xndkxy", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-12T22:01:49.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:49 smithi114 ceph-mon[25759]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:01:49.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:49 smithi114 ceph-mon[25759]: Reconfiguring daemon iscsi.foo.smithi097.xndkxy on smithi097 2024-04-12T22:01:50.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:50 smithi097 ceph-mon[24835]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-12T22:01:50.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:50 smithi097 ceph-mon[29620]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-12T22:01:50.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:50 smithi114 ceph-mon[25759]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-12T22:01:51.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:51 smithi097 ceph-mon[24835]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1001 B/s rd, 0 op/s 2024-04-12T22:01:51.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:51 smithi097 ceph-mon[29620]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1001 B/s rd, 0 op/s 2024-04-12T22:01:51.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:51 smithi114 ceph-mon[25759]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1001 B/s rd, 0 op/s 2024-04-12T22:01:53.086 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:52 smithi097 ceph-mon[24835]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:01:53.087 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:52 smithi097 ceph-mon[24835]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-04-12T22:01:53.087 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:52 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-04-12T22:01:53.087 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:52 smithi097 ceph-mon[24835]: osdmap e105: 8 total, 8 up, 8 in 2024-04-12T22:01:53.087 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:52 smithi097 ceph-mon[29620]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:01:53.087 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:52 smithi097 ceph-mon[29620]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-04-12T22:01:53.087 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:52 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-04-12T22:01:53.087 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:52 smithi097 ceph-mon[29620]: osdmap e105: 8 total, 8 up, 8 in 2024-04-12T22:01:53.087 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:01:52 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: [12/Apr/2024:22:01:52] ENGINE Bus STOPPING 2024-04-12T22:01:53.088 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:01:52 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: [12/Apr/2024:22:01:52] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-12T22:01:53.088 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:01:52 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: [12/Apr/2024:22:01:52] ENGINE Bus STOPPED 2024-04-12T22:01:53.088 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:01:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: [12/Apr/2024:22:01:53] ENGINE Bus STARTING 2024-04-12T22:01:53.152 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-mon[25759]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:01:53.152 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-mon[25759]: from='mgr.24821 172.21.15.114:0/4073315649' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-04-12T22:01:53.152 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-04-12T22:01:53.152 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-mon[25759]: osdmap e105: 8 total, 8 up, 8 in 2024-04-12T22:01:53.153 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: 2024-04-12T22:01:52.878+0000 7f7e13cb8700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi097.xndkxy ... 2024-04-12T22:01:53.153 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: Creating ceph-iscsi config... 2024-04-12T22:01:53.153 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: Write file: /var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/iscsi.foo.smithi097.xndkxy/iscsi-gateway.cfg 2024-04-12T22:01:53.153 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: Write file: /var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/iscsi.foo.smithi097.xndkxy/tcmu-runner-entrypoint.sh 2024-04-12T22:01:53.153 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: Non-zero exit code 1 from systemctl restart ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy 2024-04-12T22:01:53.153 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: systemctl: stderr Job for ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service failed because the control process exited with error code. 2024-04-12T22:01:53.153 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: systemctl: stderr See "systemctl status ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service" and "journalctl -xeu ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service" for details. 2024-04-12T22:01:53.153 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: Traceback (most recent call last): 2024-04-12T22:01:53.153 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-12T22:01:53.153 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: return _run_code(code, main_globals, None, 2024-04-12T22:01:53.153 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-12T22:01:53.153 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: exec(code, run_globals) 2024-04-12T22:01:53.153 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 5534, in 2024-04-12T22:01:53.154 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 5522, in main 2024-04-12T22:01:53.154 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 3045, in command_deploy_from 2024-04-12T22:01:53.154 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 3080, in _common_deploy 2024-04-12T22:01:53.154 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 3100, in _deploy_daemon_container 2024-04-12T22:01:53.154 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 1072, in deploy_daemon 2024-04-12T22:01:53.154 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-12T22:01:53.154 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: RuntimeError: Failed command: systemctl restart ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy: Job for ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service failed because the control process exited with error code. 2024-04-12T22:01:53.154 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: See "systemctl status ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service" and "journalctl -xeu ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service" for details. 2024-04-12T22:01:53.154 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: Traceback (most recent call last): 2024-04-12T22:01:53.154 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-04-12T22:01:53.154 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-12T22:01:53.154 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2390, in _daemon_action 2024-04-12T22:01:53.154 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-12T22:01:53.154 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: File "/usr/share/ceph/mgr/cephadm/module.py", line 747, in wait_async 2024-04-12T22:01:53.154 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: return self.event_loop.get_result(coro, timeout) 2024-04-12T22:01:53.155 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-12T22:01:53.155 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: return future.result(timeout) 2024-04-12T22:01:53.155 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-12T22:01:53.155 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: return self.__get_result() 2024-04-12T22:01:53.155 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-12T22:01:53.155 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: raise self._exception 2024-04-12T22:01:53.155 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-04-12T22:01:53.155 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: config_blobs=daemon_spec.final_config, 2024-04-12T22:01:53.155 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-04-12T22:01:53.155 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-12T22:01:53.155 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi097.xndkxy ... 2024-04-12T22:01:53.155 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: Creating ceph-iscsi config... 2024-04-12T22:01:53.155 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: Write file: /var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/iscsi.foo.smithi097.xndkxy/iscsi-gateway.cfg 2024-04-12T22:01:53.155 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: Write file: /var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/iscsi.foo.smithi097.xndkxy/tcmu-runner-entrypoint.sh 2024-04-12T22:01:53.156 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: Non-zero exit code 1 from systemctl restart ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy 2024-04-12T22:01:53.156 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: systemctl: stderr Job for ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service failed because the control process exited with error code. 2024-04-12T22:01:53.156 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: systemctl: stderr See "systemctl status ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service" and "journalctl -xeu ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service" for details. 2024-04-12T22:01:53.156 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: Traceback (most recent call last): 2024-04-12T22:01:53.156 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-12T22:01:53.156 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: return _run_code(code, main_globals, None, 2024-04-12T22:01:53.156 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-12T22:01:53.156 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: exec(code, run_globals) 2024-04-12T22:01:53.156 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 5534, in 2024-04-12T22:01:53.156 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 5522, in main 2024-04-12T22:01:53.157 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 3045, in command_deploy_from 2024-04-12T22:01:53.157 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 3080, in _common_deploy 2024-04-12T22:01:53.157 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 3100, in _deploy_daemon_container 2024-04-12T22:01:53.157 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 1072, in deploy_daemon 2024-04-12T22:01:53.157 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-12T22:01:53.157 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: RuntimeError: Failed command: systemctl restart ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy: Job for ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service failed because the control process exited with error code. 2024-04-12T22:01:53.157 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: See "systemctl status ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service" and "journalctl -xeu ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service" for details. 2024-04-12T22:01:53.157 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: 2024-04-12T22:01:52.950+0000 7f7e81645700 -1 mgr handle_mgr_map I was active but no longer am 2024-04-12T22:01:53.157 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:53 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: ignoring --setuser ceph since I am not root 2024-04-12T22:01:53.157 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:53 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: ignoring --setgroup ceph since I am not root 2024-04-12T22:01:53.157 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:53 smithi114 ceph-mgr[46922]: -- 172.21.15.114:0/3292020517 <== mon.2 v2:172.21.15.114:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55aff8a625a0 con 0x55aff8a36800 2024-04-12T22:01:53.410 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:01:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: [12/Apr/2024:22:01:53] ENGINE Serving on http://:::9283 2024-04-12T22:01:53.411 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:01:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: [12/Apr/2024:22:01:53] ENGINE Bus STARTED 2024-04-12T22:01:53.949 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:53 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: 2024-04-12T22:01:53.532+0000 7f5d96bd0200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-12T22:01:53.950 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:53 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: 2024-04-12T22:01:53.630+0000 7f5d96bd0200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-12T22:01:54.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:53 smithi114 ceph-mon[25759]: from='mgr.24821 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-04-12T22:01:54.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:53 smithi114 ceph-mon[25759]: mgrmap e27: y(active, starting, since 0.0553519s) 2024-04-12T22:01:54.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:53 smithi114 ceph-mon[25759]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-12T22:01:54.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:53 smithi114 ceph-mon[25759]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T22:01:54.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:53 smithi114 ceph-mon[25759]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T22:01:54.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:53 smithi114 ceph-mon[25759]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-12T22:01:54.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:53 smithi114 ceph-mon[25759]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T22:01:54.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:53 smithi114 ceph-mon[25759]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T22:01:54.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:53 smithi114 ceph-mon[25759]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T22:01:54.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:53 smithi114 ceph-mon[25759]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T22:01:54.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:53 smithi114 ceph-mon[25759]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T22:01:54.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:53 smithi114 ceph-mon[25759]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T22:01:54.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:53 smithi114 ceph-mon[25759]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-12T22:01:54.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:53 smithi114 ceph-mon[25759]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-12T22:01:54.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:53 smithi114 ceph-mon[25759]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-12T22:01:54.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:53 smithi114 ceph-mon[25759]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-12T22:01:54.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:53 smithi114 ceph-mon[25759]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-12T22:01:54.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:53 smithi114 ceph-mon[25759]: Manager daemon y is now available 2024-04-12T22:01:54.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:53 smithi114 ceph-mon[25759]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:01:54.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:53 smithi114 ceph-mon[25759]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:01:54.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:53 smithi114 ceph-mon[25759]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T22:01:54.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:53 smithi114 ceph-mon[25759]: from='mgr.25234 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T22:01:54.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:53 smithi114 ceph-mon[25759]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T22:01:54.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:53 smithi114 ceph-mon[25759]: from='mgr.25234 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T22:01:54.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:53 smithi097 ceph-mon[29620]: from='mgr.24821 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-04-12T22:01:54.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:53 smithi097 ceph-mon[29620]: mgrmap e27: y(active, starting, since 0.0553519s) 2024-04-12T22:01:54.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:53 smithi097 ceph-mon[29620]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-12T22:01:54.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:53 smithi097 ceph-mon[29620]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T22:01:54.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:53 smithi097 ceph-mon[29620]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T22:01:54.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:53 smithi097 ceph-mon[29620]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-12T22:01:54.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:53 smithi097 ceph-mon[29620]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T22:01:54.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:53 smithi097 ceph-mon[29620]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T22:01:54.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:53 smithi097 ceph-mon[29620]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T22:01:54.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:53 smithi097 ceph-mon[29620]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T22:01:54.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:53 smithi097 ceph-mon[29620]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T22:01:54.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:53 smithi097 ceph-mon[29620]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T22:01:54.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:53 smithi097 ceph-mon[29620]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-12T22:01:54.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:53 smithi097 ceph-mon[29620]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-12T22:01:54.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:53 smithi097 ceph-mon[29620]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-12T22:01:54.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:53 smithi097 ceph-mon[29620]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-12T22:01:54.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:53 smithi097 ceph-mon[29620]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-12T22:01:54.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:53 smithi097 ceph-mon[29620]: Manager daemon y is now available 2024-04-12T22:01:54.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:53 smithi097 ceph-mon[29620]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:01:54.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:53 smithi097 ceph-mon[29620]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:01:54.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:53 smithi097 ceph-mon[29620]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T22:01:54.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:53 smithi097 ceph-mon[29620]: from='mgr.25234 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T22:01:54.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:53 smithi097 ceph-mon[29620]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T22:01:54.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:53 smithi097 ceph-mon[29620]: from='mgr.25234 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T22:01:54.413 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:53 smithi097 ceph-mon[24835]: from='mgr.24821 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-04-12T22:01:54.413 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:53 smithi097 ceph-mon[24835]: mgrmap e27: y(active, starting, since 0.0553519s) 2024-04-12T22:01:54.413 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:53 smithi097 ceph-mon[24835]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-12T22:01:54.413 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:53 smithi097 ceph-mon[24835]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T22:01:54.413 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:53 smithi097 ceph-mon[24835]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T22:01:54.413 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:53 smithi097 ceph-mon[24835]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-12T22:01:54.413 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:53 smithi097 ceph-mon[24835]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T22:01:54.413 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:53 smithi097 ceph-mon[24835]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T22:01:54.413 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:53 smithi097 ceph-mon[24835]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T22:01:54.413 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:53 smithi097 ceph-mon[24835]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T22:01:54.413 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:53 smithi097 ceph-mon[24835]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T22:01:54.413 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:53 smithi097 ceph-mon[24835]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T22:01:54.413 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:53 smithi097 ceph-mon[24835]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-12T22:01:54.414 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:53 smithi097 ceph-mon[24835]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-12T22:01:54.414 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:53 smithi097 ceph-mon[24835]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-12T22:01:54.414 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:53 smithi097 ceph-mon[24835]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-12T22:01:54.414 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:53 smithi097 ceph-mon[24835]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-12T22:01:54.414 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:53 smithi097 ceph-mon[24835]: Manager daemon y is now available 2024-04-12T22:01:54.414 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:53 smithi097 ceph-mon[24835]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:01:54.414 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:53 smithi097 ceph-mon[24835]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:01:54.414 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:53 smithi097 ceph-mon[24835]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T22:01:54.414 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:53 smithi097 ceph-mon[24835]: from='mgr.25234 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T22:01:54.414 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:53 smithi097 ceph-mon[24835]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T22:01:54.414 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:53 smithi097 ceph-mon[24835]: from='mgr.25234 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T22:01:54.664 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:54 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: 2024-04-12T22:01:54.318+0000 7f5d96bd0200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-12T22:01:54.664 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:54 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: 2024-04-12T22:01:54.411+0000 7f5d96bd0200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-12T22:01:54.664 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:54 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: 2024-04-12T22:01:54.564+0000 7f5d96bd0200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-12T22:01:54.664 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:54 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: 2024-04-12T22:01:54.662+0000 7f5d96bd0200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-12T22:01:54.961 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:54 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: 2024-04-12T22:01:54.746+0000 7f5d96bd0200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-12T22:01:55.238 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:54 smithi114 ceph-mon[25759]: mgrmap e28: y(active, since 1.06502s) 2024-04-12T22:01:55.238 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:54 smithi114 ceph-mon[25759]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:01:55.238 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:55 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: 2024-04-12T22:01:55.026+0000 7f5d96bd0200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-12T22:01:55.238 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:55 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: 2024-04-12T22:01:55.140+0000 7f5d96bd0200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-12T22:01:55.239 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:55 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: 2024-04-12T22:01:55.236+0000 7f5d96bd0200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-12T22:01:55.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:54 smithi097 ceph-mon[24835]: mgrmap e28: y(active, since 1.06502s) 2024-04-12T22:01:55.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:54 smithi097 ceph-mon[24835]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:01:55.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:54 smithi097 ceph-mon[29620]: mgrmap e28: y(active, since 1.06502s) 2024-04-12T22:01:55.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:54 smithi097 ceph-mon[29620]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:01:55.493 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:55 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: 2024-04-12T22:01:55.345+0000 7f5d96bd0200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-12T22:01:55.878 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:55 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: 2024-04-12T22:01:55.535+0000 7f5d96bd0200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-12T22:01:56.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:55 smithi114 ceph-mon[25759]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-12T22:01:56.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:55 smithi114 ceph-mon[25759]: Cluster is now healthy 2024-04-12T22:01:56.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:55 smithi114 ceph-mon[25759]: [12/Apr/2024:22:01:55] ENGINE Bus STARTING 2024-04-12T22:01:56.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:55 smithi114 ceph-mon[25759]: [12/Apr/2024:22:01:55] ENGINE Serving on https://172.21.15.97:7150 2024-04-12T22:01:56.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:55 smithi114 ceph-mon[25759]: [12/Apr/2024:22:01:55] ENGINE Serving on http://172.21.15.97:8765 2024-04-12T22:01:56.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:55 smithi114 ceph-mon[25759]: [12/Apr/2024:22:01:55] ENGINE Bus STARTED 2024-04-12T22:01:56.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:55 smithi097 ceph-mon[24835]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-12T22:01:56.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:55 smithi097 ceph-mon[24835]: Cluster is now healthy 2024-04-12T22:01:56.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:55 smithi097 ceph-mon[24835]: [12/Apr/2024:22:01:55] ENGINE Bus STARTING 2024-04-12T22:01:56.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:55 smithi097 ceph-mon[24835]: [12/Apr/2024:22:01:55] ENGINE Serving on https://172.21.15.97:7150 2024-04-12T22:01:56.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:55 smithi097 ceph-mon[24835]: [12/Apr/2024:22:01:55] ENGINE Serving on http://172.21.15.97:8765 2024-04-12T22:01:56.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:55 smithi097 ceph-mon[24835]: [12/Apr/2024:22:01:55] ENGINE Bus STARTED 2024-04-12T22:01:56.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:55 smithi097 ceph-mon[29620]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-12T22:01:56.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:55 smithi097 ceph-mon[29620]: Cluster is now healthy 2024-04-12T22:01:56.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:55 smithi097 ceph-mon[29620]: [12/Apr/2024:22:01:55] ENGINE Bus STARTING 2024-04-12T22:01:56.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:55 smithi097 ceph-mon[29620]: [12/Apr/2024:22:01:55] ENGINE Serving on https://172.21.15.97:7150 2024-04-12T22:01:56.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:55 smithi097 ceph-mon[29620]: [12/Apr/2024:22:01:55] ENGINE Serving on http://172.21.15.97:8765 2024-04-12T22:01:56.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:55 smithi097 ceph-mon[29620]: [12/Apr/2024:22:01:55] ENGINE Bus STARTED 2024-04-12T22:01:56.978 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:56 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: 2024-04-12T22:01:56.542+0000 7f5d96bd0200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-12T22:01:56.978 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:56 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: 2024-04-12T22:01:56.683+0000 7f5d96bd0200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-12T22:01:57.243 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:01:56 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@grafana.a.service: Scheduled restart job, restart counter is at 2. 2024-04-12T22:01:57.243 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:01:56 smithi114 systemd[1]: Stopped Ceph grafana.a for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:01:57.243 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:01:57 smithi114 systemd[1]: Starting Ceph grafana.a for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:01:57.244 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:57 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: 2024-04-12T22:01:57.055+0000 7f5d96bd0200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-12T22:01:57.244 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:57 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: 2024-04-12T22:01:57.171+0000 7f5d96bd0200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-12T22:01:57.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:56 smithi114 ceph-mon[25759]: mgrmap e29: y(active, since 3s) 2024-04-12T22:01:57.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:56 smithi097 ceph-mon[24835]: mgrmap e29: y(active, since 3s) 2024-04-12T22:01:57.413 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:56 smithi097 ceph-mon[29620]: mgrmap e29: y(active, since 3s) 2024-04-12T22:01:57.545 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:57 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: 2024-04-12T22:01:57.382+0000 7f5d96bd0200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-12T22:01:57.545 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:01:57 smithi114 podman[52704]: 2024-04-12 22:01:57.148907778 +0000 UTC m=+0.022442863 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-04-12T22:01:57.545 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:01:57 smithi114 podman[52704]: 2024-04-12 22:01:57.291676246 +0000 UTC m=+0.165211307 container create 3df0adce0903e93ebfb90b567ee1e583c1bb991ad5f094808653f5d21bebeb1a (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, summary=Grafana Container configured for Ceph mgr/dashboard integration, version=8.5, maintainer=Paul Cuzner , build-date=2022-03-28T10:36:18.413762, io.openshift.tags=base rhel8, distribution-scope=public, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=ubi8-container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, release=236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, io.k8s.display-name=Red Hat Universal Base Image 8, architecture=x86_64, name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vendor=Red Hat, Inc., io.buildah.version=1.24.2, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-12T22:01:57.889 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:57 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: 2024-04-12T22:01:57.584+0000 7f5d96bd0200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-12T22:01:57.889 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:01:57 smithi114 podman[52704]: 2024-04-12 22:01:57.544359258 +0000 UTC m=+0.417894328 container init 3df0adce0903e93ebfb90b567ee1e583c1bb991ad5f094808653f5d21bebeb1a (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.24.2, vcs-type=git, io.openshift.tags=base rhel8, vendor=Red Hat, Inc., release=236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.component=ubi8-container, description=Ceph Grafana Container, io.k8s.display-name=Red Hat Universal Base Image 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=ubi8, distribution-scope=public, summary=Grafana Container configured for Ceph mgr/dashboard integration, build-date=2022-03-28T10:36:18.413762, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.openshift.expose-services=, architecture=x86_64, maintainer=Paul Cuzner , version=8.5, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI) 2024-04-12T22:01:57.889 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:01:57 smithi114 podman[52704]: 2024-04-12 22:01:57.547881635 +0000 UTC m=+0.421416698 container start 3df0adce0903e93ebfb90b567ee1e583c1bb991ad5f094808653f5d21bebeb1a (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, release=236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.component=ubi8-container, architecture=x86_64, maintainer=Paul Cuzner , io.openshift.expose-services=, io.openshift.tags=base rhel8, distribution-scope=public, description=Ceph Grafana Container, summary=Grafana Container configured for Ceph mgr/dashboard integration, build-date=2022-03-28T10:36:18.413762, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.24.2) 2024-04-12T22:01:57.889 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:01:57 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[52752]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-12T22:01:57.889 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:01:57 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[52752]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-12T22:01:57.889 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:01:57 smithi114 bash[52704]: 3df0adce0903e93ebfb90b567ee1e583c1bb991ad5f094808653f5d21bebeb1a 2024-04-12T22:01:57.890 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:01:57 smithi114 systemd[1]: Started Ceph grafana.a for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:01:57.890 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:01:57 smithi114 podman[52773]: 2024-04-12 22:01:57.662733193 +0000 UTC m=+0.077406588 container died 3df0adce0903e93ebfb90b567ee1e583c1bb991ad5f094808653f5d21bebeb1a (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, architecture=x86_64, build-date=2022-03-28T10:36:18.413762, io.openshift.expose-services=, io.openshift.tags=base rhel8, maintainer=Paul Cuzner , name=ubi8, distribution-scope=public, vcs-type=git, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.component=ubi8-container, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, release=236.1648460182, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.buildah.version=1.24.2, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, version=8.5) 2024-04-12T22:01:58.152 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:57 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: 2024-04-12T22:01:57.887+0000 7f5d96bd0200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-12T22:01:58.153 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:57 smithi114 ceph-mon[25759]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:01:58.153 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:57 smithi114 ceph-mon[25759]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:01:58.153 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:57 smithi114 ceph-mon[25759]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:01:58.410 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:57 smithi097 ceph-mon[29620]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:01:58.524 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:57 smithi097 ceph-mon[29620]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:01:58.525 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:57 smithi097 ceph-mon[29620]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:01:58.525 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:57 smithi097 ceph-mon[24835]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:01:58.525 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:57 smithi097 ceph-mon[24835]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:01:58.525 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:57 smithi097 ceph-mon[24835]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:01:58.526 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:58 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: 2024-04-12T22:01:58.151+0000 7f5d96bd0200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-12T22:01:58.526 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:58 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: 2024-04-12T22:01:58.397+0000 7f5d96bd0200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-12T22:01:58.526 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:01:58 smithi114 podman[52773]: 2024-04-12 22:01:58.348556977 +0000 UTC m=+0.763230379 container remove 3df0adce0903e93ebfb90b567ee1e583c1bb991ad5f094808653f5d21bebeb1a (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, architecture=x86_64, com.redhat.component=ubi8-container, description=Ceph Grafana Container, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-type=git, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vendor=Red Hat, Inc., version=8.5, name=ubi8, io.buildah.version=1.24.2, build-date=2022-03-28T10:36:18.413762, io.openshift.tags=base rhel8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.expose-services=, release=236.1648460182, maintainer=Paul Cuzner ) 2024-04-12T22:01:58.527 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:01:58 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-04-12T22:01:58.527 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:01:58 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-04-12T22:01:59.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:59 smithi097 ceph-mon[24835]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:01:59.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:59 smithi097 ceph-mon[24835]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:01:59.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:59 smithi097 ceph-mon[24835]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:01:59.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:59 smithi097 ceph-mon[24835]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:01:59.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:01:59 smithi097 ceph-mon[24835]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:01:59.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:59 smithi097 ceph-mon[29620]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:01:59.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:59 smithi097 ceph-mon[29620]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:01:59.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:59 smithi097 ceph-mon[29620]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:01:59.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:59 smithi097 ceph-mon[29620]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:01:59.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:01:59 smithi097 ceph-mon[29620]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:01:59.743 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:59 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: 2024-04-12T22:01:59.261+0000 7f5d96bd0200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-12T22:01:59.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:59 smithi114 ceph-mon[25759]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:01:59.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:59 smithi114 ceph-mon[25759]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:01:59.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:59 smithi114 ceph-mon[25759]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:01:59.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:59 smithi114 ceph-mon[25759]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:01:59.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:01:59 smithi114 ceph-mon[25759]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:02:00.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:59 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: 2024-04-12T22:01:59.846+0000 7f5d96bd0200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-12T22:02:00.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:01:59 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: 2024-04-12T22:01:59.947+0000 7f5d96bd0200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-12T22:02:00.493 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:00 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: 2024-04-12T22:02:00.166+0000 7f5d96bd0200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-12T22:02:00.494 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:00 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: [12/Apr/2024:22:02:00] ENGINE Bus STARTING 2024-04-12T22:02:00.494 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:00 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: CherryPy Checker: 2024-04-12T22:02:00.494 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:00 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: The Application mounted at '' has an empty config. 2024-04-12T22:02:00.494 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:00 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: 2024-04-12T22:02:00.494 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:00 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: [12/Apr/2024:22:02:00] ENGINE Serving on http://:::9283 2024-04-12T22:02:00.495 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:00 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: [12/Apr/2024:22:02:00] ENGINE Bus STARTED 2024-04-12T22:02:00.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:00 smithi097 ceph-mon[24835]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:00.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:00 smithi097 ceph-mon[24835]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:00.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:00 smithi097 ceph-mon[24835]: Standby manager daemon x started 2024-04-12T22:02:00.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:00 smithi097 ceph-mon[24835]: from='mgr.? 172.21.15.114:0/2477399193' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-12T22:02:00.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:00 smithi097 ceph-mon[24835]: from='mgr.? 172.21.15.114:0/2477399193' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-12T22:02:00.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:00 smithi097 ceph-mon[24835]: from='mgr.? 172.21.15.114:0/2477399193' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-12T22:02:00.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:00 smithi097 ceph-mon[24835]: from='mgr.? 172.21.15.114:0/2477399193' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-12T22:02:00.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:00 smithi097 ceph-mon[29620]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:00.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:00 smithi097 ceph-mon[29620]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:00.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:00 smithi097 ceph-mon[29620]: Standby manager daemon x started 2024-04-12T22:02:00.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:00 smithi097 ceph-mon[29620]: from='mgr.? 172.21.15.114:0/2477399193' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-12T22:02:00.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:00 smithi097 ceph-mon[29620]: from='mgr.? 172.21.15.114:0/2477399193' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-12T22:02:00.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:00 smithi097 ceph-mon[29620]: from='mgr.? 172.21.15.114:0/2477399193' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-12T22:02:00.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:00 smithi097 ceph-mon[29620]: from='mgr.? 172.21.15.114:0/2477399193' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-12T22:02:00.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:00 smithi114 ceph-mon[25759]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:00.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:00 smithi114 ceph-mon[25759]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:00.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:00 smithi114 ceph-mon[25759]: Standby manager daemon x started 2024-04-12T22:02:00.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:00 smithi114 ceph-mon[25759]: from='mgr.? 172.21.15.114:0/2477399193' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-12T22:02:00.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:00 smithi114 ceph-mon[25759]: from='mgr.? 172.21.15.114:0/2477399193' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-12T22:02:00.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:00 smithi114 ceph-mon[25759]: from='mgr.? 172.21.15.114:0/2477399193' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-12T22:02:00.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:00 smithi114 ceph-mon[25759]: from='mgr.? 172.21.15.114:0/2477399193' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-12T22:02:01.759 INFO:teuthology.orchestra.run.smithi097.stdout:true 2024-04-12T22:02:01.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:01 smithi097 ceph-mon[24835]: mgrmap e30: y(active, since 7s), standbys: x 2024-04-12T22:02:01.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:01 smithi097 ceph-mon[24835]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-12T22:02:01.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:01 smithi097 ceph-mon[24835]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 16 op/s 2024-04-12T22:02:01.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:01 smithi097 ceph-mon[29620]: mgrmap e30: y(active, since 7s), standbys: x 2024-04-12T22:02:01.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:01 smithi097 ceph-mon[29620]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-12T22:02:01.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:01 smithi097 ceph-mon[29620]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 16 op/s 2024-04-12T22:02:01.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:01 smithi114 ceph-mon[25759]: mgrmap e30: y(active, since 7s), standbys: x 2024-04-12T22:02:01.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:01 smithi114 ceph-mon[25759]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-12T22:02:01.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:01 smithi114 ceph-mon[25759]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 16 op/s 2024-04-12T22:02:02.476 INFO:teuthology.orchestra.run.smithi097.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T22:02:02.476 INFO:teuthology.orchestra.run.smithi097.stdout:alertmanager.a smithi097 *:9093,9094 running (58s) 5s ago 6m 13.6M - 0.23.0 ba2b418f427c 7b4c491fe0a0 2024-04-12T22:02:02.477 INFO:teuthology.orchestra.run.smithi097.stdout:grafana.a smithi114 *:3000 unknown 4s ago 5m - - 2024-04-12T22:02:02.477 INFO:teuthology.orchestra.run.smithi097.stdout:iscsi.foo.smithi097.xndkxy smithi097 unknown 5s ago 4m - - 2024-04-12T22:02:02.477 INFO:teuthology.orchestra.run.smithi097.stdout:mgr.x smithi114 *:8443,9283 running (104s) 4s ago 10m 71.4M - 19.0.0-2053-gb277feb2 668895944f0e 87e5b6de56ea 2024-04-12T22:02:02.477 INFO:teuthology.orchestra.run.smithi097.stdout:mgr.y smithi097 *:8443,9283,8765 running (24s) 5s ago 12m 475M - 19.0.0-2053-gb277feb2 668895944f0e 488170dac11f 2024-04-12T22:02:02.477 INFO:teuthology.orchestra.run.smithi097.stdout:mon.a smithi097 running (12m) 5s ago 12m 52.1M 2048M 17.2.0 e1d6a67b021e 6e618120f594 2024-04-12T22:02:02.477 INFO:teuthology.orchestra.run.smithi097.stdout:mon.b smithi114 running (10m) 4s ago 10m 47.7M 2048M 17.2.0 e1d6a67b021e 47a08e70f856 2024-04-12T22:02:02.477 INFO:teuthology.orchestra.run.smithi097.stdout:mon.c smithi097 running (10m) 5s ago 10m 47.5M 2048M 17.2.0 e1d6a67b021e 679b33a29a0c 2024-04-12T22:02:02.477 INFO:teuthology.orchestra.run.smithi097.stdout:node-exporter.a smithi097 *:9100 running (68s) 5s ago 6m 11.3M - 1.3.1 1dbe0e931976 5edff9053b37 2024-04-12T22:02:02.477 INFO:teuthology.orchestra.run.smithi097.stdout:node-exporter.b smithi114 *:9100 running (40s) 4s ago 6m 8921k - 1.3.1 1dbe0e931976 904afc45dc2f 2024-04-12T22:02:02.477 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0 smithi097 running (10m) 5s ago 10m 54.0M 2736M 17.2.0 e1d6a67b021e abac1d8884fa 2024-04-12T22:02:02.477 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1 smithi097 running (9m) 5s ago 9m 52.7M 2736M 17.2.0 e1d6a67b021e 39192562d598 2024-04-12T22:02:02.477 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2 smithi097 running (9m) 5s ago 9m 50.7M 2736M 17.2.0 e1d6a67b021e 89daf8e339c3 2024-04-12T22:02:02.477 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3 smithi097 running (8m) 5s ago 8m 51.1M 2736M 17.2.0 e1d6a67b021e 4e9a0b3862d9 2024-04-12T22:02:02.477 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4 smithi114 running (8m) 4s ago 8m 52.8M 2992M 17.2.0 e1d6a67b021e fb42c0e8e575 2024-04-12T22:02:02.478 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5 smithi114 running (8m) 4s ago 8m 49.4M 2992M 17.2.0 e1d6a67b021e 3acd4a3b61e5 2024-04-12T22:02:02.478 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6 smithi114 running (7m) 4s ago 7m 49.7M 2992M 17.2.0 e1d6a67b021e 5969baf67f41 2024-04-12T22:02:02.478 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7 smithi114 running (7m) 4s ago 7m 50.7M 2992M 17.2.0 e1d6a67b021e f68340fecf2c 2024-04-12T22:02:02.478 INFO:teuthology.orchestra.run.smithi097.stdout:prometheus.a smithi114 *:9095 running (42s) 4s ago 6m 55.1M - 2.33.4 514e6a882f6e e98d7a0f3c0c 2024-04-12T22:02:02.478 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.foo.smithi097.sggtkh smithi097 *:8000 running (5m) 5s ago 5m 71.0M - 17.2.0 e1d6a67b021e a39f9b2d8509 2024-04-12T22:02:02.478 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.foo.smithi114.ulfbyo smithi114 *:8000 running (5m) 4s ago 5m 71.6M - 17.2.0 e1d6a67b021e df79a48bdd59 2024-04-12T22:02:02.478 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.smpl.smithi097.exxkgl smithi097 *:80 running (4m) 5s ago 4m 71.2M - 17.2.0 e1d6a67b021e a036c58a0099 2024-04-12T22:02:02.478 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.smpl.smithi114.lrgvph smithi114 *:80 running (4m) 4s ago 4m 73.1M - 17.2.0 e1d6a67b021e 031a5fc24237 2024-04-12T22:02:02.902 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-12T22:02:02.902 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": { 2024-04-12T22:02:02.902 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-04-12T22:02:02.902 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:02:02.902 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": { 2024-04-12T22:02:02.902 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 2 2024-04-12T22:02:02.902 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:02:02.902 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": { 2024-04-12T22:02:02.902 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-12T22:02:02.902 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:02:02.903 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": {}, 2024-04-12T22:02:02.903 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw": { 2024-04-12T22:02:02.903 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-12T22:02:02.903 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:02:02.903 INFO:teuthology.orchestra.run.smithi097.stdout: "overall": { 2024-04-12T22:02:02.903 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 15, 2024-04-12T22:02:02.903 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 2 2024-04-12T22:02:02.903 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-12T22:02:02.903 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-12T22:02:03.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:03 smithi097 ceph-mon[29620]: from='client.15378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:02:03.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:03 smithi097 ceph-mon[29620]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:03.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:03 smithi097 ceph-mon[29620]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:03.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:03 smithi097 ceph-mon[29620]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:02:03.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:03 smithi097 ceph-mon[29620]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:02:03.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:03 smithi097 ceph-mon[29620]: Updating smithi097:/etc/ceph/ceph.conf 2024-04-12T22:02:03.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:03 smithi097 ceph-mon[29620]: Updating smithi114:/etc/ceph/ceph.conf 2024-04-12T22:02:03.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:03 smithi097 ceph-mon[29620]: from='client.15384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:02:03.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:03 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/458604992' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:02:03.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:03 smithi097 ceph-mon[24835]: from='client.15378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:02:03.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:03 smithi097 ceph-mon[24835]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:03.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:03 smithi097 ceph-mon[24835]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:03.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:03 smithi097 ceph-mon[24835]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:02:03.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:03 smithi097 ceph-mon[24835]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:02:03.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:03 smithi097 ceph-mon[24835]: Updating smithi097:/etc/ceph/ceph.conf 2024-04-12T22:02:03.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:03 smithi097 ceph-mon[24835]: Updating smithi114:/etc/ceph/ceph.conf 2024-04-12T22:02:03.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:03 smithi097 ceph-mon[24835]: from='client.15384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:02:03.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:03 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/458604992' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:02:03.294 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-12T22:02:03.294 INFO:teuthology.orchestra.run.smithi097.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a", 2024-04-12T22:02:03.294 INFO:teuthology.orchestra.run.smithi097.stdout: "in_progress": true, 2024-04-12T22:02:03.294 INFO:teuthology.orchestra.run.smithi097.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T22:02:03.294 INFO:teuthology.orchestra.run.smithi097.stdout: "services_complete": [ 2024-04-12T22:02:03.295 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr" 2024-04-12T22:02:03.295 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-04-12T22:02:03.295 INFO:teuthology.orchestra.run.smithi097.stdout: "progress": "2/23 daemons upgraded", 2024-04-12T22:02:03.295 INFO:teuthology.orchestra.run.smithi097.stdout: "message": "", 2024-04-12T22:02:03.295 INFO:teuthology.orchestra.run.smithi097.stdout: "is_paused": false 2024-04-12T22:02:03.295 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-12T22:02:03.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:03 smithi114 ceph-mon[25759]: from='client.15378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:02:03.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:03 smithi114 ceph-mon[25759]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:03.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:03 smithi114 ceph-mon[25759]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:03.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:03 smithi114 ceph-mon[25759]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:02:03.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:03 smithi114 ceph-mon[25759]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:02:03.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:03 smithi114 ceph-mon[25759]: Updating smithi097:/etc/ceph/ceph.conf 2024-04-12T22:02:03.495 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:03 smithi114 ceph-mon[25759]: Updating smithi114:/etc/ceph/ceph.conf 2024-04-12T22:02:03.495 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:03 smithi114 ceph-mon[25759]: from='client.15384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:02:03.495 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:03 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/458604992' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:02:03.726 INFO:teuthology.orchestra.run.smithi097.stdout:HEALTH_OK 2024-04-12T22:02:03.993 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:03 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[46918]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:02:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-12T22:02:04.258 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:04 smithi114 ceph-mon[25759]: Updating smithi114:/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/config/ceph.conf 2024-04-12T22:02:04.259 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:04 smithi114 ceph-mon[25759]: Updating smithi097:/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/config/ceph.conf 2024-04-12T22:02:04.259 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:04 smithi114 ceph-mon[25759]: from='client.25279 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:02:04.259 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:04 smithi114 ceph-mon[25759]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-04-12T22:02:04.259 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:04 smithi114 ceph-mon[25759]: Updating smithi097:/etc/ceph/ceph.client.admin.keyring 2024-04-12T22:02:04.259 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:04 smithi114 ceph-mon[25759]: Updating smithi114:/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/config/ceph.client.admin.keyring 2024-04-12T22:02:04.259 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:04 smithi114 ceph-mon[25759]: Updating smithi097:/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/config/ceph.client.admin.keyring 2024-04-12T22:02:04.259 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:04 smithi114 ceph-mon[25759]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-12T22:02:04.259 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:04 smithi114 ceph-mon[25759]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:04.259 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:04 smithi114 ceph-mon[25759]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:04.260 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:04 smithi114 ceph-mon[25759]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:04.260 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:04 smithi114 ceph-mon[25759]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:04.260 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:04 smithi114 ceph-mon[25759]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-12T22:02:04.260 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:04 smithi114 ceph-mon[25759]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:04.260 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:04 smithi114 ceph-mon[25759]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-12T22:02:04.260 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:04 smithi114 ceph-mon[25759]: from='client.15402 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:02:04.260 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:04 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/3786145670' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T22:02:04.410 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:04 smithi097 ceph-mon[29620]: Updating smithi114:/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/config/ceph.conf 2024-04-12T22:02:04.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:04 smithi097 ceph-mon[29620]: Updating smithi097:/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/config/ceph.conf 2024-04-12T22:02:04.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:04 smithi097 ceph-mon[29620]: from='client.25279 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:02:04.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:04 smithi097 ceph-mon[29620]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-04-12T22:02:04.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:04 smithi097 ceph-mon[29620]: Updating smithi097:/etc/ceph/ceph.client.admin.keyring 2024-04-12T22:02:04.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:04 smithi097 ceph-mon[29620]: Updating smithi114:/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/config/ceph.client.admin.keyring 2024-04-12T22:02:04.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:04 smithi097 ceph-mon[29620]: Updating smithi097:/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/config/ceph.client.admin.keyring 2024-04-12T22:02:04.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:04 smithi097 ceph-mon[29620]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-12T22:02:04.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:04 smithi097 ceph-mon[29620]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:04.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:04 smithi097 ceph-mon[29620]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:04.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:04 smithi097 ceph-mon[29620]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:04.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:04 smithi097 ceph-mon[29620]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:04.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:04 smithi097 ceph-mon[29620]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-12T22:02:04.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:04 smithi097 ceph-mon[29620]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:04.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:04 smithi097 ceph-mon[29620]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-12T22:02:04.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:04 smithi097 ceph-mon[29620]: from='client.15402 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:02:04.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:04 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/3786145670' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T22:02:04.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:04 smithi097 ceph-mon[24835]: Updating smithi114:/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/config/ceph.conf 2024-04-12T22:02:04.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:04 smithi097 ceph-mon[24835]: Updating smithi097:/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/config/ceph.conf 2024-04-12T22:02:04.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:04 smithi097 ceph-mon[24835]: from='client.25279 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:02:04.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:04 smithi097 ceph-mon[24835]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-04-12T22:02:04.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:04 smithi097 ceph-mon[24835]: Updating smithi097:/etc/ceph/ceph.client.admin.keyring 2024-04-12T22:02:04.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:04 smithi097 ceph-mon[24835]: Updating smithi114:/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/config/ceph.client.admin.keyring 2024-04-12T22:02:04.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:04 smithi097 ceph-mon[24835]: Updating smithi097:/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/config/ceph.client.admin.keyring 2024-04-12T22:02:04.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:04 smithi097 ceph-mon[24835]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-12T22:02:04.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:04 smithi097 ceph-mon[24835]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:04.413 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:04 smithi097 ceph-mon[24835]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:04.413 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:04 smithi097 ceph-mon[24835]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:04.413 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:04 smithi097 ceph-mon[24835]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:04.413 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:04 smithi097 ceph-mon[24835]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-12T22:02:04.413 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:04 smithi097 ceph-mon[24835]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:04.413 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:04 smithi097 ceph-mon[24835]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-12T22:02:04.413 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:04 smithi097 ceph-mon[24835]: from='client.15402 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:02:04.413 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:04 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/3786145670' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T22:02:05.410 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:05 smithi114 ceph-mon[25759]: Reconfiguring daemon prometheus.a on smithi114 2024-04-12T22:02:05.410 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:05 smithi114 ceph-mon[25759]: Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-12T22:02:05.410 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:05 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/1581655938' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-12T22:02:05.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:05 smithi097 ceph-mon[29620]: Reconfiguring daemon prometheus.a on smithi114 2024-04-12T22:02:05.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:05 smithi097 ceph-mon[29620]: Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-12T22:02:05.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:05 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/1581655938' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-12T22:02:05.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:05 smithi097 ceph-mon[24835]: Reconfiguring daemon prometheus.a on smithi114 2024-04-12T22:02:05.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:05 smithi097 ceph-mon[24835]: Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-12T22:02:05.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:05 smithi097 ceph-mon[24835]: from='client.? 172.21.15.97:0/1581655938' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-12T22:02:06.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:06 smithi097 ceph-mon[24835]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-12T22:02:06.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:06 smithi097 ceph-mon[29620]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-12T22:02:06.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:06 smithi114 ceph-mon[25759]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-12T22:02:06.880 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:02:06 smithi114 systemd[1]: Stopping Ceph prometheus.a for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:02:06.880 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:02:06 smithi114 bash[54049]: Error: no container with name or ID "ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus.a" found: no such container 2024-04-12T22:02:07.189 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:02:06 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[50546]: ts=2024-04-12T22:02:06.879Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-04-12T22:02:07.189 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:02:06 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[50546]: ts=2024-04-12T22:02:06.879Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-04-12T22:02:07.189 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:02:06 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[50546]: ts=2024-04-12T22:02:06.879Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-04-12T22:02:07.189 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:02:06 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[50546]: ts=2024-04-12T22:02:06.879Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-04-12T22:02:07.189 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:02:06 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[50546]: ts=2024-04-12T22:02:06.879Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-04-12T22:02:07.189 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:02:06 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[50546]: ts=2024-04-12T22:02:06.879Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-04-12T22:02:07.189 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:02:06 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[50546]: ts=2024-04-12T22:02:06.879Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-04-12T22:02:07.189 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:02:06 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[50546]: ts=2024-04-12T22:02:06.879Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-04-12T22:02:07.189 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:02:06 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[50546]: ts=2024-04-12T22:02:06.879Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-04-12T22:02:07.189 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:02:06 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[50546]: ts=2024-04-12T22:02:06.880Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-04-12T22:02:07.190 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:02:06 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[50546]: ts=2024-04-12T22:02:06.880Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-04-12T22:02:07.190 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:02:06 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[50546]: ts=2024-04-12T22:02:06.880Z caller=main.go:1066 level=info msg="See you next time!" 2024-04-12T22:02:07.190 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:02:06 smithi114 podman[54061]: 2024-04-12 22:02:06.889723249 +0000 UTC m=+0.122470731 container died e98d7a0f3c0cb95c1996341a21c0dada1e11094514b7c4a217cf14ef6074317a (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-12T22:02:07.493 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:02:07 smithi114 podman[54061]: 2024-04-12 22:02:07.37007421 +0000 UTC m=+0.602821733 container cleanup e98d7a0f3c0cb95c1996341a21c0dada1e11094514b7c4a217cf14ef6074317a (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-12T22:02:07.493 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:02:07 smithi114 bash[54061]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a 2024-04-12T22:02:07.493 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:02:07 smithi114 bash[54090]: Error: no container with name or ID "ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus.a" found: no such container 2024-04-12T22:02:07.901 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:02:07 smithi114 podman[54075]: 2024-04-12 22:02:07.671696397 +0000 UTC m=+0.779436973 container remove e98d7a0f3c0cb95c1996341a21c0dada1e11094514b7c4a217cf14ef6074317a (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-12T22:02:07.902 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:02:07 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-04-12T22:02:07.902 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:02:07 smithi114 systemd[1]: Stopped Ceph prometheus.a for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:02:07.902 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:02:07 smithi114 systemd[1]: Starting Ceph prometheus.a for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:02:08.136 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:02:07 smithi114 podman[54154]: 2024-04-12 22:02:07.805069377 +0000 UTC m=+0.018423262 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-04-12T22:02:08.136 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:02:07 smithi114 podman[54154]: 2024-04-12 22:02:07.905870955 +0000 UTC m=+0.119224828 container create 0ad5437ed63f492c7a5f2d0da45740bd3f641bc39f6f12d09a4953f220df0618 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-12T22:02:08.136 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:02:08 smithi114 podman[54154]: 2024-04-12 22:02:08.108150581 +0000 UTC m=+0.321504454 container init 0ad5437ed63f492c7a5f2d0da45740bd3f641bc39f6f12d09a4953f220df0618 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-12T22:02:08.137 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:02:08 smithi114 podman[54154]: 2024-04-12 22:02:08.110947178 +0000 UTC m=+0.324301054 container start 0ad5437ed63f492c7a5f2d0da45740bd3f641bc39f6f12d09a4953f220df0618 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-12T22:02:08.137 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:02:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:02:08.134Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-04-12T22:02:08.137 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:02:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:02:08.134Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-04-12T22:02:08.137 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:02:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:02:08.134Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-04-12T22:02:08.137 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:02:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:02:08.134Z caller=main.go:518 level=info host_details="(Linux 5.14.0-435.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024 x86_64 smithi114 (none))" 2024-04-12T22:02:08.137 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:02:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:02:08.134Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-04-12T22:02:08.137 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:02:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:02:08.134Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-04-12T22:02:08.137 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:02:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:02:08.136Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-04-12T22:02:08.175 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:02:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:02:08.136Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-04-12T22:02:08.175 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:02:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:02:08.137Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-04-12T22:02:08.175 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:02:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:02:08.139Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-04-12T22:02:08.175 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:02:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:02:08.141Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.356054ms 2024-04-12T22:02:08.175 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:02:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:02:08.141Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-04-12T22:02:08.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:08 smithi114 ceph-mon[25759]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-12T22:02:08.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:08 smithi114 ceph-mon[25759]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:02:08.494 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:02:08 smithi114 bash[54154]: 0ad5437ed63f492c7a5f2d0da45740bd3f641bc39f6f12d09a4953f220df0618 2024-04-12T22:02:08.494 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:02:08 smithi114 systemd[1]: Started Ceph prometheus.a for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:02:08.494 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:02:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:02:08.338Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=3 2024-04-12T22:02:08.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:08 smithi097 ceph-mon[24835]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-12T22:02:08.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:08 smithi097 ceph-mon[24835]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:02:08.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:08 smithi097 ceph-mon[29620]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-12T22:02:08.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:08 smithi097 ceph-mon[29620]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:02:08.760 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:08 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@grafana.a.service: Scheduled restart job, restart counter is at 3. 2024-04-12T22:02:08.760 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:08 smithi114 systemd[1]: Stopped Ceph grafana.a for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:02:08.760 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:08 smithi114 systemd[1]: Starting Ceph grafana.a for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:02:08.760 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:08 smithi114 podman[54307]: 2024-04-12 22:02:08.634078991 +0000 UTC m=+0.017265768 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-04-12T22:02:08.760 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:08 smithi114 podman[54307]: 2024-04-12 22:02:08.759423433 +0000 UTC m=+0.142610263 container create 01d93567b983d9aa796b45ce99650c3be1327fb0d395474cbf75fac5d0078d4c (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, io.k8s.display-name=Red Hat Universal Base Image 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, description=Ceph Grafana Container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base rhel8, vendor=Red Hat, Inc., maintainer=Paul Cuzner , io.buildah.version=1.24.2, io.openshift.expose-services=, version=8.5, name=ubi8, vcs-type=git, distribution-scope=public, build-date=2022-03-28T10:36:18.413762, release=236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container) 2024-04-12T22:02:08.760 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:02:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:02:08.589Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=3 2024-04-12T22:02:08.760 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:02:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:02:08.640Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=3 2024-04-12T22:02:08.760 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:02:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:02:08.640Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=3 2024-04-12T22:02:08.761 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:02:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:02:08.641Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=38.069µs wal_replay_duration=499.631838ms total_replay_duration=501.043505ms 2024-04-12T22:02:08.761 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:02:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:02:08.644Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-04-12T22:02:08.761 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:02:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:02:08.644Z caller=main.go:947 level=info msg="TSDB started" 2024-04-12T22:02:08.761 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:02:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:02:08.644Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-04-12T22:02:08.761 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:02:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:02:08.666Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=21.511166ms db_storage=908ns remote_storage=1.151µs web_handler=384ns query_engine=636ns scrape=368.83µs scrape_sd=88.975µs notify=14.12µs notify_sd=8.386µs rules=20.790513ms 2024-04-12T22:02:08.761 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:02:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:02:08.666Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-04-12T22:02:09.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:09 smithi114 ceph-mon[25759]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:09.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:09 smithi114 ceph-mon[25759]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:09.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:09 smithi114 ceph-mon[25759]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-12T22:02:09.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:09 smithi114 ceph-mon[25759]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-12T22:02:09.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:09 smithi114 ceph-mon[25759]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:02:09.494 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:09 smithi114 podman[54307]: 2024-04-12 22:02:09.028439135 +0000 UTC m=+0.411625906 container init 01d93567b983d9aa796b45ce99650c3be1327fb0d395474cbf75fac5d0078d4c (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Grafana Container configured for Ceph mgr/dashboard integration, release=236.1648460182, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, description=Ceph Grafana Container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.buildah.version=1.24.2, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.tags=base rhel8, com.redhat.component=ubi8-container, name=ubi8, architecture=x86_64, version=8.5, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-03-28T10:36:18.413762, maintainer=Paul Cuzner , distribution-scope=public, io.openshift.expose-services=) 2024-04-12T22:02:09.494 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:09 smithi114 podman[54307]: 2024-04-12 22:02:09.031103851 +0000 UTC m=+0.414290625 container start 01d93567b983d9aa796b45ce99650c3be1327fb0d395474cbf75fac5d0078d4c (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, io.buildah.version=1.24.2, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , description=Ceph Grafana Container, vcs-type=git, io.openshift.tags=base rhel8, com.redhat.component=ubi8-container, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, name=ubi8, build-date=2022-03-28T10:36:18.413762, release=236.1648460182, vendor=Red Hat, Inc., io.openshift.expose-services=, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, architecture=x86_64) 2024-04-12T22:02:09.494 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:09 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[54336]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-12T22:02:09.494 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:09 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[54336]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-12T22:02:09.494 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:09 smithi114 bash[54307]: 01d93567b983d9aa796b45ce99650c3be1327fb0d395474cbf75fac5d0078d4c 2024-04-12T22:02:09.494 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:09 smithi114 systemd[1]: Started Ceph grafana.a for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:02:09.494 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:09 smithi114 podman[54364]: 2024-04-12 22:02:09.13628698 +0000 UTC m=+0.072253875 container died 01d93567b983d9aa796b45ce99650c3be1327fb0d395474cbf75fac5d0078d4c (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, io.buildah.version=1.24.2, architecture=x86_64, description=Ceph Grafana Container, release=236.1648460182, summary=Grafana Container configured for Ceph mgr/dashboard integration, vendor=Red Hat, Inc., distribution-scope=public, name=ubi8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, vcs-type=git, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, maintainer=Paul Cuzner , build-date=2022-03-28T10:36:18.413762, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=ubi8-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.openshift.tags=base rhel8) 2024-04-12T22:02:09.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:09 smithi097 ceph-mon[24835]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:09.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:09 smithi097 ceph-mon[24835]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:09.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:09 smithi097 ceph-mon[24835]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-12T22:02:09.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:09 smithi097 ceph-mon[24835]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-12T22:02:09.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:09 smithi097 ceph-mon[24835]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:02:09.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:09 smithi097 ceph-mon[29620]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:09.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:09 smithi097 ceph-mon[29620]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:09.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:09 smithi097 ceph-mon[29620]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-12T22:02:09.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:09 smithi097 ceph-mon[29620]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-12T22:02:09.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:09 smithi097 ceph-mon[29620]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:02:10.011 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:09 smithi114 podman[54364]: 2024-04-12 22:02:09.748955284 +0000 UTC m=+0.684922224 container remove 01d93567b983d9aa796b45ce99650c3be1327fb0d395474cbf75fac5d0078d4c (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, com.redhat.component=ubi8-container, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-type=git, maintainer=Paul Cuzner , vendor=Red Hat, Inc., build-date=2022-03-28T10:36:18.413762, io.openshift.expose-services=, architecture=x86_64, release=236.1648460182, io.buildah.version=1.24.2, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, name=ubi8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, description=Ceph Grafana Container, version=8.5, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI) 2024-04-12T22:02:10.011 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:09 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-04-12T22:02:10.011 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:09 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-04-12T22:02:10.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:10 smithi114 ceph-mon[25759]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-12T22:02:10.562 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:10 smithi097 ceph-mon[24835]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-12T22:02:10.562 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:10 smithi097 ceph-mon[29620]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-12T22:02:11.469 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:11 smithi114 ceph-mon[25759]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T22:02:11.469 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:11 smithi114 ceph-mon[25759]: from='mgr.25234 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T22:02:11.469 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:11 smithi114 ceph-mon[25759]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T22:02:11.469 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:11 smithi114 ceph-mon[25759]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:02:11.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:11 smithi097 ceph-mon[24835]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T22:02:11.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:11 smithi097 ceph-mon[24835]: from='mgr.25234 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T22:02:11.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:11 smithi097 ceph-mon[24835]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T22:02:11.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:11 smithi097 ceph-mon[24835]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:02:11.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:11 smithi097 ceph-mon[29620]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T22:02:11.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:11 smithi097 ceph-mon[29620]: from='mgr.25234 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T22:02:11.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:11 smithi097 ceph-mon[29620]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T22:02:11.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:11 smithi097 ceph-mon[29620]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:02:12.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:12 smithi114 ceph-mon[25759]: Upgrade: Updating mgr.x 2024-04-12T22:02:12.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:12 smithi114 ceph-mon[25759]: Deploying daemon mgr.x on smithi114 2024-04-12T22:02:12.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:12 smithi114 ceph-mon[25759]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 502 B/s rd, 0 op/s 2024-04-12T22:02:12.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:12 smithi097 ceph-mon[24835]: Upgrade: Updating mgr.x 2024-04-12T22:02:12.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:12 smithi097 ceph-mon[24835]: Deploying daemon mgr.x on smithi114 2024-04-12T22:02:12.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:12 smithi097 ceph-mon[24835]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 502 B/s rd, 0 op/s 2024-04-12T22:02:12.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:12 smithi097 ceph-mon[29620]: Upgrade: Updating mgr.x 2024-04-12T22:02:12.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:12 smithi097 ceph-mon[29620]: Deploying daemon mgr.x on smithi114 2024-04-12T22:02:12.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:12 smithi097 ceph-mon[29620]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 502 B/s rd, 0 op/s 2024-04-12T22:02:13.115 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:12 smithi114 systemd[1]: Stopping Ceph mgr.x for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:02:13.427 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:13 smithi114 podman[54598]: 2024-04-12 22:02:13.11495082 +0000 UTC m=+0.124554723 container died 87e5b6de56ea9c82e7a157f02a63bf1d91ee5a65e8a2ffb0476e414026bb097c (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-b277feb, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , ceph=True) 2024-04-12T22:02:13.743 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:13 smithi114 podman[54598]: 2024-04-12 22:02:13.508700888 +0000 UTC m=+0.518304799 container cleanup 87e5b6de56ea9c82e7a157f02a63bf1d91ee5a65e8a2ffb0476e414026bb097c (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-b277feb, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0) 2024-04-12T22:02:13.743 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:13 smithi114 bash[54598]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x 2024-04-12T22:02:13.910 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:02:13 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[59315]: level=warn ts=2024-04-12T22:02:13.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi114.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.114:8443: connect: connection refused" 2024-04-12T22:02:14.076 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:13 smithi114 podman[54609]: 2024-04-12 22:02:13.751739794 +0000 UTC m=+0.634892052 container remove 87e5b6de56ea9c82e7a157f02a63bf1d91ee5a65e8a2ffb0476e414026bb097c (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-b277feb, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-12T22:02:14.076 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:13 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-04-12T22:02:14.076 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:13 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@mgr.x.service: Failed with result 'exit-code'. 2024-04-12T22:02:14.076 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:13 smithi114 systemd[1]: Stopped Ceph mgr.x for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:02:14.077 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:13 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@mgr.x.service: Consumed 27.571s CPU time. 2024-04-12T22:02:14.359 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:14 smithi114 ceph-mon[25759]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 502 B/s rd, 0 op/s 2024-04-12T22:02:14.359 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:14 smithi114 systemd[1]: Starting Ceph mgr.x for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:02:14.359 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:14 smithi114 podman[54740]: 2024-04-12 22:02:14.191670373 +0000 UTC m=+0.018690697 image pull 668895944f0ec64b67a5b7564961d2756f2966e5d4b9338516300dbec9984951 quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a 2024-04-12T22:02:14.631 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:14 smithi114 podman[54740]: 2024-04-12 22:02:14.359157017 +0000 UTC m=+0.186177331 container create b8f8df3bf9280c4d8c3b4c7c82fab2d551196d40bb88b4d57ecb9a66961097bc (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-b277feb, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2) 2024-04-12T22:02:14.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:14 smithi097 ceph-mon[24835]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 502 B/s rd, 0 op/s 2024-04-12T22:02:14.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:14 smithi097 ceph-mon[29620]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 502 B/s rd, 0 op/s 2024-04-12T22:02:14.993 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:14 smithi114 podman[54740]: 2024-04-12 22:02:14.630815963 +0000 UTC m=+0.457836277 container init b8f8df3bf9280c4d8c3b4c7c82fab2d551196d40bb88b4d57ecb9a66961097bc (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-b277feb, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409) 2024-04-12T22:02:14.993 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:14 smithi114 podman[54740]: 2024-04-12 22:02:14.633501059 +0000 UTC m=+0.460521384 container start b8f8df3bf9280c4d8c3b4c7c82fab2d551196d40bb88b4d57ecb9a66961097bc (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-b277feb, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240409) 2024-04-12T22:02:14.993 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:14 smithi114 ceph-mgr[54759]: -- 172.21.15.114:0/3373354702 <== mon.2 v2:172.21.15.114:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5614c25965a0 con 0x5614c256ac00 2024-04-12T22:02:14.994 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:14 smithi114 bash[54740]: b8f8df3bf9280c4d8c3b4c7c82fab2d551196d40bb88b4d57ecb9a66961097bc 2024-04-12T22:02:14.994 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:14 smithi114 systemd[1]: Started Ceph mgr.x for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:02:15.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:15 smithi114 ceph-mon[25759]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:02:15.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:15 smithi114 ceph-mon[25759]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:15.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:15 smithi114 ceph-mon[25759]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:15.493 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:15 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: 2024-04-12T22:02:15.199+0000 7fe4937e7200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-12T22:02:15.493 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:15 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: 2024-04-12T22:02:15.307+0000 7fe4937e7200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-12T22:02:15.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:15 smithi097 ceph-mon[24835]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:02:15.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:15 smithi097 ceph-mon[24835]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:15.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:15 smithi097 ceph-mon[24835]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:15.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:15 smithi097 ceph-mon[29620]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:02:15.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:15 smithi097 ceph-mon[29620]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:15.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:15 smithi097 ceph-mon[29620]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:16.223 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:16 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: 2024-04-12T22:02:16.000+0000 7fe4937e7200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-12T22:02:16.223 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:16 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: 2024-04-12T22:02:16.093+0000 7fe4937e7200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-12T22:02:16.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:16 smithi114 ceph-mon[25759]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:02:16.493 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:16 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: 2024-04-12T22:02:16.244+0000 7fe4937e7200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-12T22:02:16.494 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:16 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: 2024-04-12T22:02:16.340+0000 7fe4937e7200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-12T22:02:16.494 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:16 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: 2024-04-12T22:02:16.425+0000 7fe4937e7200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-12T22:02:16.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:16 smithi097 ceph-mon[24835]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:02:16.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:16 smithi097 ceph-mon[29620]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:02:16.817 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:16 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: 2024-04-12T22:02:16.705+0000 7fe4937e7200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-12T22:02:17.215 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:16 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: 2024-04-12T22:02:16.816+0000 7fe4937e7200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-12T22:02:17.215 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:16 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: 2024-04-12T22:02:16.916+0000 7fe4937e7200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-12T22:02:17.215 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:17 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: 2024-04-12T22:02:17.023+0000 7fe4937e7200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-12T22:02:17.216 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:17 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: 2024-04-12T22:02:17.214+0000 7fe4937e7200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-12T22:02:18.368 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:18 smithi114 ceph-mon[25759]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:02:18.368 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:18 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: 2024-04-12T22:02:18.235+0000 7fe4937e7200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-12T22:02:18.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:18 smithi097 ceph-mon[24835]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:02:18.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:18 smithi097 ceph-mon[29620]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:02:19.054 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:18 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: 2024-04-12T22:02:18.366+0000 7fe4937e7200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-12T22:02:19.054 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:18 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: 2024-04-12T22:02:18.743+0000 7fe4937e7200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-12T22:02:19.054 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:18 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: 2024-04-12T22:02:18.840+0000 7fe4937e7200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-12T22:02:19.493 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:19 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: 2024-04-12T22:02:19.052+0000 7fe4937e7200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-12T22:02:19.493 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:19 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: 2024-04-12T22:02:19.251+0000 7fe4937e7200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-12T22:02:19.660 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:02:19 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:02:19] "GET /metrics HTTP/1.1" 200 38239 "" "Prometheus/2.33.4" 2024-04-12T22:02:19.836 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:19 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: 2024-04-12T22:02:19.558+0000 7fe4937e7200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-12T22:02:19.836 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:19 smithi114 ceph-mon[25759]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:19.836 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:19 smithi114 ceph-mon[25759]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:19.836 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:19 smithi114 ceph-mon[25759]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:02:20.096 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:19 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: 2024-04-12T22:02:19.835+0000 7fe4937e7200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-12T22:02:20.096 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:19 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@grafana.a.service: Scheduled restart job, restart counter is at 4. 2024-04-12T22:02:20.096 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:19 smithi114 systemd[1]: Stopped Ceph grafana.a for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:02:20.097 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:20 smithi114 systemd[1]: Starting Ceph grafana.a for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:02:20.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:19 smithi097 ceph-mon[24835]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:20.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:19 smithi097 ceph-mon[24835]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:20.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:19 smithi097 ceph-mon[24835]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:02:20.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:19 smithi097 ceph-mon[29620]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:20.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:19 smithi097 ceph-mon[29620]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:20.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:19 smithi097 ceph-mon[29620]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:02:20.365 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:20 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: 2024-04-12T22:02:20.094+0000 7fe4937e7200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-12T22:02:20.366 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:20 smithi114 podman[55667]: 2024-04-12 22:02:20.230425758 +0000 UTC m=+0.107120185 container create f12e5c3fcabbc6ab595d4ace8192f0afddc723afb80fa0dfbfc16f5c48df12cf (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base rhel8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, distribution-scope=public, name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, io.buildah.version=1.24.2, io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , architecture=x86_64, version=8.5, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, release=236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, build-date=2022-03-28T10:36:18.413762, description=Ceph Grafana Container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.openshift.expose-services=, vendor=Red Hat, Inc.) 2024-04-12T22:02:20.366 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:20 smithi114 podman[55667]: 2024-04-12 22:02:20.142916685 +0000 UTC m=+0.019611138 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-04-12T22:02:20.859 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:20 smithi114 podman[55667]: 2024-04-12 22:02:20.5084542 +0000 UTC m=+0.385148649 container init f12e5c3fcabbc6ab595d4ace8192f0afddc723afb80fa0dfbfc16f5c48df12cf (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, summary=Grafana Container configured for Ceph mgr/dashboard integration, maintainer=Paul Cuzner , com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, distribution-scope=public, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.component=ubi8-container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-03-28T10:36:18.413762, description=Ceph Grafana Container, io.openshift.tags=base rhel8, io.openshift.expose-services=, io.buildah.version=1.24.2, vendor=Red Hat, Inc., version=8.5, release=236.1648460182, name=ubi8) 2024-04-12T22:02:20.859 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:20 smithi114 podman[55667]: 2024-04-12 22:02:20.512141056 +0000 UTC m=+0.388835484 container start f12e5c3fcabbc6ab595d4ace8192f0afddc723afb80fa0dfbfc16f5c48df12cf (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, io.openshift.expose-services=, io.buildah.version=1.24.2, distribution-scope=public, vendor=Red Hat, Inc., name=ubi8, io.openshift.tags=base rhel8, architecture=x86_64, com.redhat.component=ubi8-container, maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, release=236.1648460182, io.k8s.display-name=Red Hat Universal Base Image 8, build-date=2022-03-28T10:36:18.413762, version=8.5, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, description=Ceph Grafana Container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-04-12T22:02:20.859 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:20 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[55698]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-12T22:02:20.859 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:20 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[55698]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-12T22:02:20.859 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:20 smithi114 bash[55667]: f12e5c3fcabbc6ab595d4ace8192f0afddc723afb80fa0dfbfc16f5c48df12cf 2024-04-12T22:02:20.859 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:20 smithi114 systemd[1]: Started Ceph grafana.a for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:02:21.145 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:20 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: 2024-04-12T22:02:20.959+0000 7fe4937e7200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-12T22:02:21.145 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:20 smithi114 ceph-mon[25759]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:21.145 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:20 smithi114 ceph-mon[25759]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:21.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:20 smithi097 ceph-mon[24835]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:21.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:20 smithi097 ceph-mon[24835]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:21.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:20 smithi097 ceph-mon[29620]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:21.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:20 smithi097 ceph-mon[29620]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:21.432 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:21 smithi114 podman[55727]: 2024-04-12 22:02:21.144693949 +0000 UTC m=+0.598549805 container remove f12e5c3fcabbc6ab595d4ace8192f0afddc723afb80fa0dfbfc16f5c48df12cf (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, build-date=2022-03-28T10:36:18.413762, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.24.2, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vendor=Red Hat, Inc., description=Ceph Grafana Container, release=236.1648460182, io.openshift.tags=base rhel8, io.openshift.expose-services=, architecture=x86_64, summary=Grafana Container configured for Ceph mgr/dashboard integration, maintainer=Paul Cuzner , distribution-scope=public, name=ubi8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git) 2024-04-12T22:02:21.432 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:21 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-04-12T22:02:21.433 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:21 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-04-12T22:02:21.744 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: 2024-04-12T22:02:21.559+0000 7fe4937e7200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-12T22:02:21.744 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: 2024-04-12T22:02:21.663+0000 7fe4937e7200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-12T22:02:22.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:21 smithi097 ceph-mon[24835]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:02:22.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:21 smithi097 ceph-mon[29620]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:02:22.233 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:21 smithi114 ceph-mon[25759]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:02:22.233 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: 2024-04-12T22:02:21.894+0000 7fe4937e7200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-12T22:02:22.234 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: [12/Apr/2024:22:02:21] ENGINE Bus STARTING 2024-04-12T22:02:22.234 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:22 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: CherryPy Checker: 2024-04-12T22:02:22.234 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:22 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: The Application mounted at '' has an empty config. 2024-04-12T22:02:22.234 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:22 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: 2024-04-12T22:02:22.234 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:22 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: [12/Apr/2024:22:02:22] ENGINE Serving on http://:::9283 2024-04-12T22:02:22.234 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:22 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: [12/Apr/2024:22:02:22] ENGINE Bus STARTED 2024-04-12T22:02:23.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:22 smithi097 ceph-mon[24835]: Standby manager daemon x restarted 2024-04-12T22:02:23.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:22 smithi097 ceph-mon[24835]: Standby manager daemon x started 2024-04-12T22:02:23.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:22 smithi097 ceph-mon[24835]: from='mgr.? 172.21.15.114:0/988742303' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-12T22:02:23.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:22 smithi097 ceph-mon[24835]: from='mgr.? 172.21.15.114:0/988742303' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-12T22:02:23.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:22 smithi097 ceph-mon[24835]: from='mgr.? 172.21.15.114:0/988742303' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-12T22:02:23.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:22 smithi097 ceph-mon[24835]: from='mgr.? 172.21.15.114:0/988742303' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-12T22:02:23.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:22 smithi097 ceph-mon[29620]: Standby manager daemon x restarted 2024-04-12T22:02:23.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:22 smithi097 ceph-mon[29620]: Standby manager daemon x started 2024-04-12T22:02:23.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:22 smithi097 ceph-mon[29620]: from='mgr.? 172.21.15.114:0/988742303' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-12T22:02:23.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:22 smithi097 ceph-mon[29620]: from='mgr.? 172.21.15.114:0/988742303' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-12T22:02:23.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:22 smithi097 ceph-mon[29620]: from='mgr.? 172.21.15.114:0/988742303' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-12T22:02:23.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:22 smithi097 ceph-mon[29620]: from='mgr.? 172.21.15.114:0/988742303' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-12T22:02:23.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:22 smithi114 ceph-mon[25759]: Standby manager daemon x restarted 2024-04-12T22:02:23.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:22 smithi114 ceph-mon[25759]: Standby manager daemon x started 2024-04-12T22:02:23.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:22 smithi114 ceph-mon[25759]: from='mgr.? 172.21.15.114:0/988742303' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-12T22:02:23.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:22 smithi114 ceph-mon[25759]: from='mgr.? 172.21.15.114:0/988742303' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-12T22:02:23.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:22 smithi114 ceph-mon[25759]: from='mgr.? 172.21.15.114:0/988742303' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-12T22:02:23.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:22 smithi114 ceph-mon[25759]: from='mgr.? 172.21.15.114:0/988742303' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-12T22:02:23.873 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:02:23 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[59315]: level=error ts=2024-04-12T22:02:23.509Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=7 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi114.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.114:8443: connect: connection refused" 2024-04-12T22:02:24.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:23 smithi097 ceph-mon[24835]: mgrmap e31: y(active, since 29s), standbys: x 2024-04-12T22:02:24.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:23 smithi097 ceph-mon[24835]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:02:24.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:23 smithi097 ceph-mon[24835]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:02:24.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:23 smithi097 ceph-mon[29620]: mgrmap e31: y(active, since 29s), standbys: x 2024-04-12T22:02:24.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:23 smithi097 ceph-mon[29620]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:02:24.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:23 smithi097 ceph-mon[29620]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:02:24.202 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:23 smithi114 ceph-mon[25759]: mgrmap e31: y(active, since 29s), standbys: x 2024-04-12T22:02:24.202 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:23 smithi114 ceph-mon[25759]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:02:24.202 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:23 smithi114 ceph-mon[25759]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:02:25.323 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:25 smithi097 ceph-mon[29620]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:25.323 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:25 smithi097 ceph-mon[29620]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:02:25.324 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:25 smithi097 ceph-mon[29620]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:25.324 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:25 smithi097 ceph-mon[29620]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:02:25.324 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:25 smithi097 ceph-mon[29620]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:02:25.324 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:25 smithi097 ceph-mon[29620]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-12T22:02:25.324 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:25 smithi097 ceph-mon[29620]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-12T22:02:25.324 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:25 smithi097 ceph-mon[29620]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:25.324 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:25 smithi097 ceph-mon[29620]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:02:25.324 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:25 smithi097 ceph-mon[29620]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:02:25.324 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:25 smithi097 ceph-mon[29620]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:25.324 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:25 smithi097 ceph-mon[29620]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-04-12T22:02:25.324 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:25 smithi097 ceph-mon[29620]: from='mgr.25234 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-04-12T22:02:25.324 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:25 smithi097 ceph-mon[29620]: from='mgr.25234 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-04-12T22:02:25.324 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:25 smithi097 ceph-mon[29620]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-04-12T22:02:25.325 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:25 smithi097 ceph-mon[29620]: from='mgr.25234 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-04-12T22:02:25.325 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:25 smithi097 ceph-mon[29620]: from='mgr.25234 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-04-12T22:02:25.325 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:25 smithi097 ceph-mon[29620]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-12T22:02:25.325 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:25 smithi097 ceph-mon[29620]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-04-12T22:02:25.325 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:25 smithi097 ceph-mon[24835]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:25.325 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:25 smithi097 ceph-mon[24835]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:02:25.325 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:25 smithi097 ceph-mon[24835]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:25.325 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:25 smithi097 ceph-mon[24835]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:02:25.325 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:25 smithi097 ceph-mon[24835]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:02:25.325 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:25 smithi097 ceph-mon[24835]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-12T22:02:25.326 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:25 smithi097 ceph-mon[24835]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-12T22:02:25.326 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:25 smithi097 ceph-mon[24835]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:25.326 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:25 smithi097 ceph-mon[24835]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:02:25.326 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:25 smithi097 ceph-mon[24835]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:02:25.326 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:25 smithi097 ceph-mon[24835]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:25.326 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:25 smithi097 ceph-mon[24835]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-04-12T22:02:25.326 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:25 smithi097 ceph-mon[24835]: from='mgr.25234 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-04-12T22:02:25.326 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:25 smithi097 ceph-mon[24835]: from='mgr.25234 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-04-12T22:02:25.326 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:25 smithi097 ceph-mon[24835]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-04-12T22:02:25.326 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:25 smithi097 ceph-mon[24835]: from='mgr.25234 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-04-12T22:02:25.326 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:25 smithi097 ceph-mon[24835]: from='mgr.25234 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-04-12T22:02:25.326 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:25 smithi097 ceph-mon[24835]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-12T22:02:25.326 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:25 smithi097 ceph-mon[24835]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-04-12T22:02:25.644 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:25 smithi114 ceph-mon[25759]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:25.644 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:25 smithi114 ceph-mon[25759]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:02:25.644 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:25 smithi114 ceph-mon[25759]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:25.644 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:25 smithi114 ceph-mon[25759]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:02:25.644 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:25 smithi114 ceph-mon[25759]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:02:25.644 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:25 smithi114 ceph-mon[25759]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-12T22:02:25.644 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:25 smithi114 ceph-mon[25759]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-12T22:02:25.644 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:25 smithi114 ceph-mon[25759]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:25.644 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:25 smithi114 ceph-mon[25759]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:02:25.644 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:25 smithi114 ceph-mon[25759]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:02:25.644 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:25 smithi114 ceph-mon[25759]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:25.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:25 smithi114 ceph-mon[25759]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-04-12T22:02:25.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:25 smithi114 ceph-mon[25759]: from='mgr.25234 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-04-12T22:02:25.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:25 smithi114 ceph-mon[25759]: from='mgr.25234 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-04-12T22:02:25.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:25 smithi114 ceph-mon[25759]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-04-12T22:02:25.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:25 smithi114 ceph-mon[25759]: from='mgr.25234 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-04-12T22:02:25.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:25 smithi114 ceph-mon[25759]: from='mgr.25234 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-04-12T22:02:25.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:25 smithi114 ceph-mon[25759]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-12T22:02:25.645 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:25 smithi114 ceph-mon[25759]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-04-12T22:02:26.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:26 smithi097 ceph-mon[24835]: Upgrade: Setting container_image for all mgr 2024-04-12T22:02:26.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:26 smithi097 ceph-mon[24835]: Upgrade: It appears safe to stop mon.a 2024-04-12T22:02:26.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:26 smithi097 ceph-mon[24835]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:26.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:26 smithi097 ceph-mon[24835]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T22:02:26.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:26 smithi097 ceph-mon[24835]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T22:02:26.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:26 smithi097 ceph-mon[24835]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:02:26.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:26 smithi097 ceph-mon[29620]: Upgrade: Setting container_image for all mgr 2024-04-12T22:02:26.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:26 smithi097 ceph-mon[29620]: Upgrade: It appears safe to stop mon.a 2024-04-12T22:02:26.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:26 smithi097 ceph-mon[29620]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:26.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:26 smithi097 ceph-mon[29620]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T22:02:26.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:26 smithi097 ceph-mon[29620]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T22:02:26.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:26 smithi097 ceph-mon[29620]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:02:26.539 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:26 smithi114 ceph-mon[25759]: Upgrade: Setting container_image for all mgr 2024-04-12T22:02:26.539 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:26 smithi114 ceph-mon[25759]: Upgrade: It appears safe to stop mon.a 2024-04-12T22:02:26.539 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:26 smithi114 ceph-mon[25759]: from='mgr.25234 ' entity='mgr.y' 2024-04-12T22:02:26.539 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:26 smithi114 ceph-mon[25759]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T22:02:26.539 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:26 smithi114 ceph-mon[25759]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T22:02:26.539 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:26 smithi114 ceph-mon[25759]: from='mgr.25234 172.21.15.97:0/3881160064' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:02:27.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:27 smithi097 ceph-mon[29620]: Upgrade: Updating mon.a 2024-04-12T22:02:27.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:27 smithi097 ceph-mon[29620]: Deploying daemon mon.a on smithi097 2024-04-12T22:02:27.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:27 smithi097 ceph-mon[29620]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-12T22:02:27.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:27 smithi097 ceph-mon[24835]: Upgrade: Updating mon.a 2024-04-12T22:02:27.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:27 smithi097 ceph-mon[24835]: Deploying daemon mon.a on smithi097 2024-04-12T22:02:27.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:27 smithi097 ceph-mon[24835]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-12T22:02:27.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:27 smithi114 ceph-mon[25759]: Upgrade: Updating mon.a 2024-04-12T22:02:27.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:27 smithi114 ceph-mon[25759]: Deploying daemon mon.a on smithi097 2024-04-12T22:02:27.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:27 smithi114 ceph-mon[25759]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-12T22:02:28.265 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:27 smithi097 systemd[1]: Stopping Ceph mon.a for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:02:28.265 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:28 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mon-a[24831]: 2024-04-12T22:02:28.213+0000 7f70f2715700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-12T22:02:28.265 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:28 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mon-a[24831]: 2024-04-12T22:02:28.213+0000 7f70f2715700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-12T22:02:28.574 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:28 smithi097 podman[64682]: 2024-04-12 22:02:28.264782194 +0000 UTC m=+0.154548688 container died 6e618120f594da25a122688ce6b75f57044039180e09d197734437dbae4e5f7d (image=quay.io/ceph/ceph:v17.2.0, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mon-a, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, ceph=True, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, release=754, architecture=x86_64, name=centos-stream, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-12T22:02:28.882 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:28 smithi097 podman[64682]: 2024-04-12 22:02:28.672143824 +0000 UTC m=+0.561910317 container cleanup 6e618120f594da25a122688ce6b75f57044039180e09d197734437dbae4e5f7d (image=quay.io/ceph/ceph:v17.2.0, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mon-a, maintainer=Guillaume Abrioux , vcs-type=git, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., RELEASE=HEAD, architecture=x86_64, GIT_CLEAN=True, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, version=8) 2024-04-12T22:02:28.882 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:28 smithi097 bash[64682]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mon-a 2024-04-12T22:02:28.994 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:02:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:02:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"smithi097:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"172.21.15.114:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-12T22:02:29.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:28 smithi097 podman[64696]: 2024-04-12 22:02:28.881740066 +0000 UTC m=+0.616793896 container remove 6e618120f594da25a122688ce6b75f57044039180e09d197734437dbae4e5f7d (image=quay.io/ceph/ceph:v17.2.0, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mon-a, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, distribution-scope=public, GIT_CLEAN=True, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, vcs-type=git, com.redhat.component=centos-stream-container) 2024-04-12T22:02:29.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-12T22:02:29.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 systemd[1]: Stopped Ceph mon.a for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:02:29.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@mon.a.service: Consumed 13.575s CPU time. 2024-04-12T22:02:29.450 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:02:29] "GET /metrics HTTP/1.1" 200 38233 "" "Prometheus/2.33.4" 2024-04-12T22:02:29.450 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 systemd[1]: Starting Ceph mon.a for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:02:29.451 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 podman[64834]: 2024-04-12 22:02:29.312419922 +0000 UTC m=+0.017642694 image pull 668895944f0ec64b67a5b7564961d2756f2966e5d4b9338516300dbec9984951 quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a 2024-04-12T22:02:29.713 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 podman[64834]: 2024-04-12 22:02:29.44973083 +0000 UTC m=+0.154953589 container create 3cfa16d3a85ce6f8f5277bd983073020a2cfac36820302600bf776922f7ee50a (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mon-a, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b277feb, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-12T22:02:29.713 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 podman[64834]: 2024-04-12 22:02:29.686163859 +0000 UTC m=+0.391386773 container init 3cfa16d3a85ce6f8f5277bd983073020a2cfac36820302600bf776922f7ee50a (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mon-a, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-b277feb) 2024-04-12T22:02:29.714 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 podman[64834]: 2024-04-12 22:02:29.689515884 +0000 UTC m=+0.394738651 container start 3cfa16d3a85ce6f8f5277bd983073020a2cfac36820302600bf776922f7ee50a (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mon-a, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-b277feb, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-12T22:02:30.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: set uid:gid to 167:167 (ceph:ceph) 2024-04-12T22:02:30.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev), process ceph-mon, pid 2 2024-04-12T22:02:30.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: pidfile_write: ignore empty --pid-file 2024-04-12T22:02:30.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: load: jerasure load: lrc 2024-04-12T22:02:30.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: RocksDB version: 7.9.2 2024-04-12T22:02:30.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Git sha 0 2024-04-12T22:02:30.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Compile date 2024-04-09 21:18:54 2024-04-12T22:02:30.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: DB SUMMARY 2024-04-12T22:02:30.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: DB Session ID: JKX61QHGF3OJK7D50I3A 2024-04-12T22:02:30.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: CURRENT file: CURRENT 2024-04-12T22:02:30.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: IDENTITY file: IDENTITY 2024-04-12T22:02:30.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: MANIFEST file: MANIFEST-000015 size: 980 Bytes 2024-04-12T22:02:30.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000030.sst 2024-04-12T22:02:30.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000028.log size: 3646104 ; 2024-04-12T22:02:30.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.error_if_exists: 0 2024-04-12T22:02:30.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.create_if_missing: 0 2024-04-12T22:02:30.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.paranoid_checks: 1 2024-04-12T22:02:30.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-12T22:02:30.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-12T22:02:30.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-12T22:02:30.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.env: 0x55fc4685dc60 2024-04-12T22:02:30.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.fs: PosixFileSystem 2024-04-12T22:02:30.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.info_log: 0x55fc483efae0 2024-04-12T22:02:30.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.max_file_opening_threads: 16 2024-04-12T22:02:30.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.statistics: (nil) 2024-04-12T22:02:30.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.use_fsync: 0 2024-04-12T22:02:30.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.max_log_file_size: 0 2024-04-12T22:02:30.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-12T22:02:30.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-12T22:02:30.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.keep_log_file_num: 1000 2024-04-12T22:02:30.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.recycle_log_file_num: 0 2024-04-12T22:02:30.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.allow_fallocate: 1 2024-04-12T22:02:30.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.allow_mmap_reads: 0 2024-04-12T22:02:30.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.allow_mmap_writes: 0 2024-04-12T22:02:30.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.use_direct_reads: 0 2024-04-12T22:02:30.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-12T22:02:30.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.create_missing_column_families: 0 2024-04-12T22:02:30.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.db_log_dir: 2024-04-12T22:02:30.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.wal_dir: 2024-04-12T22:02:30.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-12T22:02:30.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-12T22:02:30.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-12T22:02:30.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-12T22:02:30.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-12T22:02:30.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-12T22:02:30.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.advise_random_on_open: 1 2024-04-12T22:02:30.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.db_write_buffer_size: 0 2024-04-12T22:02:30.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.write_buffer_manager: 0x55fc483c2820 2024-04-12T22:02:30.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-12T22:02:30.166 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-12T22:02:30.166 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-12T22:02:30.166 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.rate_limiter: (nil) 2024-04-12T22:02:30.166 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-12T22:02:30.166 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.wal_recovery_mode: 2 2024-04-12T22:02:30.166 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.enable_thread_tracking: 0 2024-04-12T22:02:30.166 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.enable_pipelined_write: 0 2024-04-12T22:02:30.166 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.unordered_write: 0 2024-04-12T22:02:30.166 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-12T22:02:30.166 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-12T22:02:30.166 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-12T22:02:30.167 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-12T22:02:30.167 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.row_cache: None 2024-04-12T22:02:30.167 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.wal_filter: None 2024-04-12T22:02:30.167 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-12T22:02:30.167 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.allow_ingest_behind: 0 2024-04-12T22:02:30.167 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.two_write_queues: 0 2024-04-12T22:02:30.167 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.manual_wal_flush: 0 2024-04-12T22:02:30.167 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.wal_compression: 0 2024-04-12T22:02:30.167 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.atomic_flush: 0 2024-04-12T22:02:30.167 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-12T22:02:30.167 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-12T22:02:30.167 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-12T22:02:30.168 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.log_readahead_size: 0 2024-04-12T22:02:30.168 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-12T22:02:30.168 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.best_efforts_recovery: 0 2024-04-12T22:02:30.168 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-12T22:02:30.168 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-12T22:02:30.168 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.allow_data_in_errors: 0 2024-04-12T22:02:30.168 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.db_host_id: __hostname__ 2024-04-12T22:02:30.168 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-12T22:02:30.170 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.max_background_jobs: 2 2024-04-12T22:02:30.170 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.max_background_compactions: -1 2024-04-12T22:02:30.170 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.max_subcompactions: 1 2024-04-12T22:02:30.170 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-12T22:02:30.170 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-12T22:02:30.170 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-12T22:02:30.170 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.max_total_wal_size: 0 2024-04-12T22:02:30.170 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-12T22:02:30.170 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-12T22:02:30.170 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-12T22:02:30.170 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-12T22:02:30.171 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.max_open_files: -1 2024-04-12T22:02:30.171 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.bytes_per_sync: 0 2024-04-12T22:02:30.171 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-12T22:02:30.171 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-12T22:02:30.171 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.compaction_readahead_size: 0 2024-04-12T22:02:30.171 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.max_background_flushes: -1 2024-04-12T22:02:30.171 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Compression algorithms supported: 2024-04-12T22:02:30.171 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-12T22:02:30.171 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: kZSTD supported: 0 2024-04-12T22:02:30.171 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: kXpressCompression supported: 0 2024-04-12T22:02:30.171 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: kLZ4HCCompression supported: 1 2024-04-12T22:02:30.171 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: kZlibCompression supported: 1 2024-04-12T22:02:30.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: kSnappyCompression supported: 1 2024-04-12T22:02:30.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: kLZ4Compression supported: 1 2024-04-12T22:02:30.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: kBZip2Compression supported: 0 2024-04-12T22:02:30.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-12T22:02:30.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-12T22:02:30.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 2024-04-12T22:02:30.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-12T22:02:30.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-12T22:02:30.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.merge_operator: 2024-04-12T22:02:30.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.compaction_filter: None 2024-04-12T22:02:30.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.compaction_filter_factory: None 2024-04-12T22:02:30.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.sst_partitioner_factory: None 2024-04-12T22:02:30.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-12T22:02:30.172 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-12T22:02:30.173 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55fc483efbe0) 2024-04-12T22:02:30.173 INFO:journalctl@ceph.mon.a.smithi097.stdout: cache_index_and_filter_blocks: 1 2024-04-12T22:02:30.173 INFO:journalctl@ceph.mon.a.smithi097.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-12T22:02:30.173 INFO:journalctl@ceph.mon.a.smithi097.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-12T22:02:30.173 INFO:journalctl@ceph.mon.a.smithi097.stdout: pin_top_level_index_and_filter: 1 2024-04-12T22:02:30.173 INFO:journalctl@ceph.mon.a.smithi097.stdout: index_type: 0 2024-04-12T22:02:30.173 INFO:journalctl@ceph.mon.a.smithi097.stdout: data_block_index_type: 0 2024-04-12T22:02:30.173 INFO:journalctl@ceph.mon.a.smithi097.stdout: index_shortening: 1 2024-04-12T22:02:30.173 INFO:journalctl@ceph.mon.a.smithi097.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-12T22:02:30.173 INFO:journalctl@ceph.mon.a.smithi097.stdout: checksum: 4 2024-04-12T22:02:30.173 INFO:journalctl@ceph.mon.a.smithi097.stdout: no_block_cache: 0 2024-04-12T22:02:30.173 INFO:journalctl@ceph.mon.a.smithi097.stdout: block_cache: 0x55fc483f3090 2024-04-12T22:02:30.173 INFO:journalctl@ceph.mon.a.smithi097.stdout: block_cache_name: BinnedLRUCache 2024-04-12T22:02:30.174 INFO:journalctl@ceph.mon.a.smithi097.stdout: block_cache_options: 2024-04-12T22:02:30.174 INFO:journalctl@ceph.mon.a.smithi097.stdout: capacity : 536870912 2024-04-12T22:02:30.174 INFO:journalctl@ceph.mon.a.smithi097.stdout: num_shard_bits : 4 2024-04-12T22:02:30.174 INFO:journalctl@ceph.mon.a.smithi097.stdout: strict_capacity_limit : 0 2024-04-12T22:02:30.174 INFO:journalctl@ceph.mon.a.smithi097.stdout: high_pri_pool_ratio: 0.000 2024-04-12T22:02:30.174 INFO:journalctl@ceph.mon.a.smithi097.stdout: block_cache_compressed: (nil) 2024-04-12T22:02:30.174 INFO:journalctl@ceph.mon.a.smithi097.stdout: persistent_cache: (nil) 2024-04-12T22:02:30.174 INFO:journalctl@ceph.mon.a.smithi097.stdout: block_size: 4096 2024-04-12T22:02:30.174 INFO:journalctl@ceph.mon.a.smithi097.stdout: block_size_deviation: 10 2024-04-12T22:02:30.174 INFO:journalctl@ceph.mon.a.smithi097.stdout: block_restart_interval: 16 2024-04-12T22:02:30.174 INFO:journalctl@ceph.mon.a.smithi097.stdout: index_block_restart_interval: 1 2024-04-12T22:02:30.175 INFO:journalctl@ceph.mon.a.smithi097.stdout: metadata_block_size: 4096 2024-04-12T22:02:30.175 INFO:journalctl@ceph.mon.a.smithi097.stdout: partition_filters: 0 2024-04-12T22:02:30.175 INFO:journalctl@ceph.mon.a.smithi097.stdout: use_delta_encoding: 1 2024-04-12T22:02:30.175 INFO:journalctl@ceph.mon.a.smithi097.stdout: filter_policy: bloomfilter 2024-04-12T22:02:30.175 INFO:journalctl@ceph.mon.a.smithi097.stdout: whole_key_filtering: 1 2024-04-12T22:02:30.175 INFO:journalctl@ceph.mon.a.smithi097.stdout: verify_compression: 0 2024-04-12T22:02:30.175 INFO:journalctl@ceph.mon.a.smithi097.stdout: read_amp_bytes_per_bit: 0 2024-04-12T22:02:30.175 INFO:journalctl@ceph.mon.a.smithi097.stdout: format_version: 5 2024-04-12T22:02:30.175 INFO:journalctl@ceph.mon.a.smithi097.stdout: enable_index_compression: 1 2024-04-12T22:02:30.175 INFO:journalctl@ceph.mon.a.smithi097.stdout: block_align: 0 2024-04-12T22:02:30.175 INFO:journalctl@ceph.mon.a.smithi097.stdout: max_auto_readahead_size: 262144 2024-04-12T22:02:30.175 INFO:journalctl@ceph.mon.a.smithi097.stdout: prepopulate_block_cache: 0 2024-04-12T22:02:30.175 INFO:journalctl@ceph.mon.a.smithi097.stdout: initial_auto_readahead_size: 8192 2024-04-12T22:02:30.176 INFO:journalctl@ceph.mon.a.smithi097.stdout: num_file_reads_for_auto_readahead: 2 2024-04-12T22:02:30.176 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.write_buffer_size: 33554432 2024-04-12T22:02:30.176 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.max_write_buffer_number: 2 2024-04-12T22:02:30.176 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.compression: NoCompression 2024-04-12T22:02:30.176 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.bottommost_compression: Disabled 2024-04-12T22:02:30.176 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.prefix_extractor: nullptr 2024-04-12T22:02:30.176 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-12T22:02:30.176 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.num_levels: 7 2024-04-12T22:02:30.176 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-12T22:02:30.176 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-12T22:02:30.176 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-12T22:02:30.176 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-12T22:02:30.177 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-12T22:02:30.177 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-12T22:02:30.177 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-12T22:02:30.177 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-12T22:02:30.177 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-12T22:02:30.177 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-12T22:02:30.177 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-12T22:02:30.177 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-12T22:02:30.177 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-12T22:02:30.177 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.compression_opts.level: 32767 2024-04-12T22:02:30.177 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.compression_opts.strategy: 0 2024-04-12T22:02:30.178 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-12T22:02:30.178 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-12T22:02:30.178 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-12T22:02:30.178 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-12T22:02:30.178 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.compression_opts.enabled: false 2024-04-12T22:02:30.178 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-12T22:02:30.178 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-12T22:02:30.178 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-12T22:02:30.178 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-12T22:02:30.178 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.target_file_size_base: 67108864 2024-04-12T22:02:30.178 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-12T22:02:30.179 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-12T22:02:30.179 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-12T22:02:30.179 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-12T22:02:30.179 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-12T22:02:30.179 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-12T22:02:30.179 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-12T22:02:30.179 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-12T22:02:30.179 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-12T22:02:30.179 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-12T22:02:30.179 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-12T22:02:30.179 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-12T22:02:30.179 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-12T22:02:30.179 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-12T22:02:30.179 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.arena_block_size: 1048576 2024-04-12T22:02:30.180 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-12T22:02:30.180 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-12T22:02:30.180 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.disable_auto_compactions: 0 2024-04-12T22:02:30.180 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-12T22:02:30.180 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-12T22:02:30.180 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-12T22:02:30.180 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-12T22:02:30.180 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-12T22:02:30.180 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-12T22:02:30.180 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-12T22:02:30.180 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-12T22:02:30.180 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-12T22:02:30.180 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-12T22:02:30.181 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.table_properties_collectors: 2024-04-12T22:02:30.181 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.inplace_update_support: 0 2024-04-12T22:02:30.181 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-12T22:02:30.181 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-12T22:02:30.181 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-12T22:02:30.181 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-12T22:02:30.181 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.bloom_locality: 0 2024-04-12T22:02:30.181 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.max_successive_merges: 0 2024-04-12T22:02:30.181 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-12T22:02:30.181 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.paranoid_file_checks: 0 2024-04-12T22:02:30.181 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.force_consistency_checks: 1 2024-04-12T22:02:30.181 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.report_bg_io_stats: 0 2024-04-12T22:02:30.181 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.ttl: 2592000 2024-04-12T22:02:30.181 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-12T22:02:30.182 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-12T22:02:30.182 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-12T22:02:30.182 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.enable_blob_files: false 2024-04-12T22:02:30.182 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.min_blob_size: 0 2024-04-12T22:02:30.182 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.blob_file_size: 268435456 2024-04-12T22:02:30.182 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-12T22:02:30.182 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-12T22:02:30.182 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-12T22:02:30.182 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-12T22:02:30.182 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-12T22:02:30.182 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.blob_file_starting_level: 0 2024-04-12T22:02:30.182 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-12T22:02:30.182 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 30.sst 2024-04-12T22:02:30.183 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-12T22:02:30.183 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 32, last_sequence is 12061, log_number is 28,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-12T22:02:30.183 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 28 2024-04-12T22:02:30.183 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 6a0567cf-40f3-4edb-bf08-5bb1d06c76be 2024-04-12T22:02:30.183 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712959349719541, "job": 1, "event": "recovery_started", "wal_files": [28]} 2024-04-12T22:02:30.183 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #28 mode 2 2024-04-12T22:02:30.183 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712959349729084, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 33, "file_size": 3352198, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12062, "largest_seqno": 13721, "table_properties": {"data_size": 3344344, "index_size": 4978, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1861, "raw_key_size": 16898, "raw_average_key_size": 23, "raw_value_size": 3328642, "raw_average_value_size": 4635, "num_data_blocks": 226, "num_entries": 718, "num_filter_entries": 718, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1712959349, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "6a0567cf-40f3-4edb-bf08-5bb1d06c76be", "db_session_id": "JKX61QHGF3OJK7D50I3A", "orig_file_number": 33, "seqno_to_time_mapping": "N/A"}} 2024-04-12T22:02:30.183 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712959349729184, "job": 1, "event": "recovery_finished"} 2024-04-12T22:02:30.183 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: [db/version_set.cc:5047] Creating manifest 35 2024-04-12T22:02:30.183 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-12T22:02:30.183 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000028.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-12T22:02:30.183 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55fc484e4000 2024-04-12T22:02:30.184 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: DB pointer 0x55fc484ce000 2024-04-12T22:02:30.184 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-12T22:02:30.184 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-12T22:02:30.184 INFO:journalctl@ceph.mon.a.smithi097.stdout: ** DB Stats ** 2024-04-12T22:02:30.184 INFO:journalctl@ceph.mon.a.smithi097.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-12T22:02:30.184 INFO:journalctl@ceph.mon.a.smithi097.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-12T22:02:30.184 INFO:journalctl@ceph.mon.a.smithi097.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-12T22:02:30.184 INFO:journalctl@ceph.mon.a.smithi097.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-12T22:02:30.184 INFO:journalctl@ceph.mon.a.smithi097.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-12T22:02:30.184 INFO:journalctl@ceph.mon.a.smithi097.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-12T22:02:30.184 INFO:journalctl@ceph.mon.a.smithi097.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-12T22:02:30.184 INFO:journalctl@ceph.mon.a.smithi097.stdout: 2024-04-12T22:02:30.185 INFO:journalctl@ceph.mon.a.smithi097.stdout: ** Compaction Stats [default] ** 2024-04-12T22:02:30.185 INFO:journalctl@ceph.mon.a.smithi097.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-12T22:02:30.185 INFO:journalctl@ceph.mon.a.smithi097.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-12T22:02:30.185 INFO:journalctl@ceph.mon.a.smithi097.stdout: L0 1/0 3.20 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 729.9 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-04-12T22:02:30.185 INFO:journalctl@ceph.mon.a.smithi097.stdout: L6 1/0 7.58 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-12T22:02:30.185 INFO:journalctl@ceph.mon.a.smithi097.stdout: Sum 2/0 10.77 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 729.9 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-04-12T22:02:30.185 INFO:journalctl@ceph.mon.a.smithi097.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 729.9 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-04-12T22:02:30.185 INFO:journalctl@ceph.mon.a.smithi097.stdout: 2024-04-12T22:02:30.185 INFO:journalctl@ceph.mon.a.smithi097.stdout: ** Compaction Stats [default] ** 2024-04-12T22:02:30.185 INFO:journalctl@ceph.mon.a.smithi097.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-12T22:02:30.185 INFO:journalctl@ceph.mon.a.smithi097.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-12T22:02:30.185 INFO:journalctl@ceph.mon.a.smithi097.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 729.9 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-04-12T22:02:30.186 INFO:journalctl@ceph.mon.a.smithi097.stdout: 2024-04-12T22:02:30.186 INFO:journalctl@ceph.mon.a.smithi097.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-12T22:02:30.186 INFO:journalctl@ceph.mon.a.smithi097.stdout: 2024-04-12T22:02:30.186 INFO:journalctl@ceph.mon.a.smithi097.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-12T22:02:30.186 INFO:journalctl@ceph.mon.a.smithi097.stdout: Flush(GB): cumulative 0.003, interval 0.003 2024-04-12T22:02:30.186 INFO:journalctl@ceph.mon.a.smithi097.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-12T22:02:30.186 INFO:journalctl@ceph.mon.a.smithi097.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-12T22:02:30.186 INFO:journalctl@ceph.mon.a.smithi097.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-12T22:02:30.186 INFO:journalctl@ceph.mon.a.smithi097.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-12T22:02:30.186 INFO:journalctl@ceph.mon.a.smithi097.stdout: Cumulative compaction: 0.00 GB write, 232.23 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-12T22:02:30.186 INFO:journalctl@ceph.mon.a.smithi097.stdout: Interval compaction: 0.00 GB write, 232.23 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-12T22:02:30.186 INFO:journalctl@ceph.mon.a.smithi097.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-12T22:02:30.187 INFO:journalctl@ceph.mon.a.smithi097.stdout: Block cache BinnedLRUCache@0x55fc483f3090#2 capacity: 512.00 MB usage: 7.11 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-04-12T22:02:30.187 INFO:journalctl@ceph.mon.a.smithi097.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,2.03 KB,0.00038743%) IndexBlock(1,5.08 KB,0.000968575%) Misc(1,0.00 KB,0%) 2024-04-12T22:02:30.187 INFO:journalctl@ceph.mon.a.smithi097.stdout: 2024-04-12T22:02:30.187 INFO:journalctl@ceph.mon.a.smithi097.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-12T22:02:30.187 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: starting mon.a rank 0 at public addrs [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] at bind addrs [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 80bf7680-f916-11ee-bc8d-c7b262605968 2024-04-12T22:02:30.187 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: mon.a@-1(???) e3 preinit fsid 80bf7680-f916-11ee-bc8d-c7b262605968 2024-04-12T22:02:30.187 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: mon.a@-1(???).mds e1 new map 2024-04-12T22:02:30.187 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: mon.a@-1(???).mds e1 print_map 2024-04-12T22:02:30.187 INFO:journalctl@ceph.mon.a.smithi097.stdout: e1 2024-04-12T22:02:30.187 INFO:journalctl@ceph.mon.a.smithi097.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T22:02:30.187 INFO:journalctl@ceph.mon.a.smithi097.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T22:02:30.187 INFO:journalctl@ceph.mon.a.smithi097.stdout: legacy client fscid: -1 2024-04-12T22:02:30.188 INFO:journalctl@ceph.mon.a.smithi097.stdout: 2024-04-12T22:02:30.188 INFO:journalctl@ceph.mon.a.smithi097.stdout: No filesystems configured 2024-04-12T22:02:30.188 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: mon.a@-1(???).osd e105 crush map has features 3314933000854323200, adjusting msgr requires 2024-04-12T22:02:30.188 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: mon.a@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-04-12T22:02:30.188 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: mon.a@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-04-12T22:02:30.188 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: mon.a@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-04-12T22:02:30.188 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 ceph-mon[64853]: mon.a@-1(???).paxosservice(auth 1..22) refresh upgraded, format 0 -> 3 2024-04-12T22:02:30.188 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 bash[64834]: 3cfa16d3a85ce6f8f5277bd983073020a2cfac36820302600bf776922f7ee50a 2024-04-12T22:02:30.188 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:29 smithi097 systemd[1]: Started Ceph mon.a for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:02:31.503 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:31 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@grafana.a.service: Scheduled restart job, restart counter is at 5. 2024-04-12T22:02:31.503 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:31 smithi114 systemd[1]: Stopped Ceph grafana.a for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:02:31.503 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:31 smithi114 systemd[1]: Starting Ceph grafana.a for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:02:31.503 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:31 smithi114 podman[55998]: 2024-04-12 22:02:31.381196267 +0000 UTC m=+0.017528252 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-04-12T22:02:31.817 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:31 smithi114 podman[55998]: 2024-04-12 22:02:31.502982393 +0000 UTC m=+0.139314353 container create 1072b63601d58a672bd2b057c7952b731362ea13222581455d5bd75db96bd9db (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, version=8.5, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, description=Ceph Grafana Container, com.redhat.component=ubi8-container, name=ubi8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.24.2, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.openshift.expose-services=, build-date=2022-03-28T10:36:18.413762, vendor=Red Hat, Inc., release=236.1648460182, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, io.openshift.tags=base rhel8, architecture=x86_64) 2024-04-12T22:02:31.817 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:31 smithi114 podman[55998]: 2024-04-12 22:02:31.707511483 +0000 UTC m=+0.343843450 container init 1072b63601d58a672bd2b057c7952b731362ea13222581455d5bd75db96bd9db (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, io.buildah.version=1.24.2, com.redhat.component=ubi8-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, maintainer=Paul Cuzner , name=ubi8, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, release=236.1648460182, io.openshift.tags=base rhel8, version=8.5, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, architecture=x86_64, build-date=2022-03-28T10:36:18.413762, vendor=Red Hat, Inc., distribution-scope=public, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, description=Ceph Grafana Container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56) 2024-04-12T22:02:31.817 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:31 smithi114 podman[55998]: 2024-04-12 22:02:31.710036755 +0000 UTC m=+0.346368716 container start 1072b63601d58a672bd2b057c7952b731362ea13222581455d5bd75db96bd9db (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, build-date=2022-03-28T10:36:18.413762, maintainer=Paul Cuzner , vcs-type=git, version=8.5, description=Ceph Grafana Container, name=ubi8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public, vendor=Red Hat, Inc., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.24.2, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=236.1648460182, io.openshift.tags=base rhel8, com.redhat.component=ubi8-container, io.openshift.expose-services=) 2024-04-12T22:02:31.817 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:31 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[56013]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-12T22:02:31.817 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:31 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[56013]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-12T22:02:31.817 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:31 smithi114 bash[55998]: 1072b63601d58a672bd2b057c7952b731362ea13222581455d5bd75db96bd9db 2024-04-12T22:02:32.243 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:31 smithi114 systemd[1]: Started Ceph grafana.a for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:02:32.243 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:31 smithi114 podman[56029]: 2024-04-12 22:02:31.844706012 +0000 UTC m=+0.097237192 container died 1072b63601d58a672bd2b057c7952b731362ea13222581455d5bd75db96bd9db (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, vendor=Red Hat, Inc., version=8.5, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.24.2, io.openshift.tags=base rhel8, io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , build-date=2022-03-28T10:36:18.413762, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, name=ubi8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=236.1648460182, distribution-scope=public, summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, com.redhat.component=ubi8-container, io.openshift.expose-services=, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container) 2024-04-12T22:02:32.743 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:32 smithi114 podman[56029]: 2024-04-12 22:02:32.377680172 +0000 UTC m=+0.630211344 container remove 1072b63601d58a672bd2b057c7952b731362ea13222581455d5bd75db96bd9db (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, io.buildah.version=1.24.2, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, io.openshift.tags=base rhel8, release=236.1648460182, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, version=8.5, name=ubi8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container, build-date=2022-03-28T10:36:18.413762, distribution-scope=public, vcs-type=git, com.redhat.component=ubi8-container, maintainer=Paul Cuzner , architecture=x86_64, vendor=Red Hat, Inc.) 2024-04-12T22:02:32.743 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:32 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-04-12T22:02:32.743 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:32 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-04-12T22:02:33.160 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:02:32 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: ignoring --setuser ceph since I am not root 2024-04-12T22:02:33.161 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:02:32 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: ignoring --setgroup ceph since I am not root 2024-04-12T22:02:33.161 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:02:32 smithi097 ceph-mgr[60559]: -- 172.21.15.97:0/396535736 <== mon.1 v2:172.21.15.97:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5585c6f6a5a0 con 0x5585c6f3f000 2024-04-12T22:02:33.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:32 smithi097 ceph-mon[29620]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 561 B/s rd, 0 op/s 2024-04-12T22:02:33.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:32 smithi097 ceph-mon[29620]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-12T22:02:33.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:32 smithi097 ceph-mon[29620]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-12T22:02:33.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:32 smithi097 ceph-mon[29620]: mon.a calling monitor election 2024-04-12T22:02:33.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:32 smithi097 ceph-mon[29620]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-12T22:02:33.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:32 smithi097 ceph-mon[29620]: monmap epoch 3 2024-04-12T22:02:33.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:32 smithi097 ceph-mon[29620]: fsid 80bf7680-f916-11ee-bc8d-c7b262605968 2024-04-12T22:02:33.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:32 smithi097 ceph-mon[29620]: last_changed 2024-04-12T21:51:25.246349+0000 2024-04-12T22:02:33.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:32 smithi097 ceph-mon[29620]: created 2024-04-12T21:49:45.534140+0000 2024-04-12T22:02:33.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:32 smithi097 ceph-mon[29620]: min_mon_release 17 (quincy) 2024-04-12T22:02:33.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:32 smithi097 ceph-mon[29620]: election_strategy: 1 2024-04-12T22:02:33.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:32 smithi097 ceph-mon[29620]: 0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2024-04-12T22:02:33.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:32 smithi097 ceph-mon[29620]: 1: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2024-04-12T22:02:33.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:32 smithi097 ceph-mon[29620]: 2: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-04-12T22:02:33.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:32 smithi097 ceph-mon[29620]: fsmap 2024-04-12T22:02:33.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:32 smithi097 ceph-mon[29620]: osdmap e105: 8 total, 8 up, 8 in 2024-04-12T22:02:33.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:32 smithi097 ceph-mon[29620]: mgrmap e31: y(active, since 39s), standbys: x 2024-04-12T22:02:33.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:32 smithi097 ceph-mon[29620]: Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2024-04-12T22:02:33.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:32 smithi097 ceph-mon[29620]: [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-04-12T22:02:33.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:32 smithi097 ceph-mon[29620]: daemon iscsi.foo.smithi097.xndkxy on smithi097 is in unknown state 2024-04-12T22:02:33.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:32 smithi097 ceph-mon[29620]: daemon grafana.a on smithi114 is in unknown state 2024-04-12T22:02:33.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:32 smithi097 ceph-mon[29620]: from='mgr.25234 ' entity='' 2024-04-12T22:02:33.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:32 smithi097 ceph-mon[29620]: mgrmap e32: y(active, since 39s), standbys: x 2024-04-12T22:02:33.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:32 smithi097 ceph-mon[64853]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 561 B/s rd, 0 op/s 2024-04-12T22:02:33.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:32 smithi097 ceph-mon[64853]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-12T22:02:33.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:32 smithi097 ceph-mon[64853]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-12T22:02:33.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:32 smithi097 ceph-mon[64853]: mon.a calling monitor election 2024-04-12T22:02:33.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:32 smithi097 ceph-mon[64853]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-12T22:02:33.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:32 smithi097 ceph-mon[64853]: monmap epoch 3 2024-04-12T22:02:33.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:32 smithi097 ceph-mon[64853]: fsid 80bf7680-f916-11ee-bc8d-c7b262605968 2024-04-12T22:02:33.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:32 smithi097 ceph-mon[64853]: last_changed 2024-04-12T21:51:25.246349+0000 2024-04-12T22:02:33.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:32 smithi097 ceph-mon[64853]: created 2024-04-12T21:49:45.534140+0000 2024-04-12T22:02:33.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:32 smithi097 ceph-mon[64853]: min_mon_release 17 (quincy) 2024-04-12T22:02:33.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:32 smithi097 ceph-mon[64853]: election_strategy: 1 2024-04-12T22:02:33.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:32 smithi097 ceph-mon[64853]: 0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2024-04-12T22:02:33.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:32 smithi097 ceph-mon[64853]: 1: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2024-04-12T22:02:33.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:32 smithi097 ceph-mon[64853]: 2: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-04-12T22:02:33.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:32 smithi097 ceph-mon[64853]: fsmap 2024-04-12T22:02:33.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:32 smithi097 ceph-mon[64853]: osdmap e105: 8 total, 8 up, 8 in 2024-04-12T22:02:33.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:32 smithi097 ceph-mon[64853]: mgrmap e31: y(active, since 39s), standbys: x 2024-04-12T22:02:33.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:32 smithi097 ceph-mon[64853]: Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2024-04-12T22:02:33.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:32 smithi097 ceph-mon[64853]: [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-04-12T22:02:33.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:32 smithi097 ceph-mon[64853]: daemon iscsi.foo.smithi097.xndkxy on smithi097 is in unknown state 2024-04-12T22:02:33.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:32 smithi097 ceph-mon[64853]: daemon grafana.a on smithi114 is in unknown state 2024-04-12T22:02:33.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:32 smithi097 ceph-mon[64853]: from='mgr.25234 ' entity='' 2024-04-12T22:02:33.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:32 smithi097 ceph-mon[64853]: mgrmap e32: y(active, since 39s), standbys: x 2024-04-12T22:02:33.243 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:32 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: ignoring --setuser ceph since I am not root 2024-04-12T22:02:33.243 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:32 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: ignoring --setgroup ceph since I am not root 2024-04-12T22:02:33.243 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:32 smithi114 ceph-mgr[54759]: -- 172.21.15.114:0/3387048177 <== mon.2 v2:172.21.15.114:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5589b3eb65a0 con 0x5589b3e8ac00 2024-04-12T22:02:33.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:32 smithi114 ceph-mon[25759]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 561 B/s rd, 0 op/s 2024-04-12T22:02:33.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:32 smithi114 ceph-mon[25759]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-12T22:02:33.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:32 smithi114 ceph-mon[25759]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-12T22:02:33.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:32 smithi114 ceph-mon[25759]: mon.a calling monitor election 2024-04-12T22:02:33.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:32 smithi114 ceph-mon[25759]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-12T22:02:33.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:32 smithi114 ceph-mon[25759]: monmap epoch 3 2024-04-12T22:02:33.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:32 smithi114 ceph-mon[25759]: fsid 80bf7680-f916-11ee-bc8d-c7b262605968 2024-04-12T22:02:33.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:32 smithi114 ceph-mon[25759]: last_changed 2024-04-12T21:51:25.246349+0000 2024-04-12T22:02:33.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:32 smithi114 ceph-mon[25759]: created 2024-04-12T21:49:45.534140+0000 2024-04-12T22:02:33.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:32 smithi114 ceph-mon[25759]: min_mon_release 17 (quincy) 2024-04-12T22:02:33.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:32 smithi114 ceph-mon[25759]: election_strategy: 1 2024-04-12T22:02:33.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:32 smithi114 ceph-mon[25759]: 0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2024-04-12T22:02:33.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:32 smithi114 ceph-mon[25759]: 1: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2024-04-12T22:02:33.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:32 smithi114 ceph-mon[25759]: 2: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-04-12T22:02:33.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:32 smithi114 ceph-mon[25759]: fsmap 2024-04-12T22:02:33.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:32 smithi114 ceph-mon[25759]: osdmap e105: 8 total, 8 up, 8 in 2024-04-12T22:02:33.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:32 smithi114 ceph-mon[25759]: mgrmap e31: y(active, since 39s), standbys: x 2024-04-12T22:02:33.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:32 smithi114 ceph-mon[25759]: Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2024-04-12T22:02:33.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:32 smithi114 ceph-mon[25759]: [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-04-12T22:02:33.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:32 smithi114 ceph-mon[25759]: daemon iscsi.foo.smithi097.xndkxy on smithi097 is in unknown state 2024-04-12T22:02:33.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:32 smithi114 ceph-mon[25759]: daemon grafana.a on smithi114 is in unknown state 2024-04-12T22:02:33.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:32 smithi114 ceph-mon[25759]: from='mgr.25234 ' entity='' 2024-04-12T22:02:33.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:32 smithi114 ceph-mon[25759]: mgrmap e32: y(active, since 39s), standbys: x 2024-04-12T22:02:33.660 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:02:33 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:02:33.340+0000 7fe4598af200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-12T22:02:33.660 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:02:33 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:02:33.434+0000 7fe4598af200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-12T22:02:33.661 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:02:33 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[59315]: level=warn ts=2024-04-12T22:02:33.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi097.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.97:8443: connect: connection refused" 2024-04-12T22:02:33.661 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:02:33 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[59315]: level=warn ts=2024-04-12T22:02:33.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi114.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.114:8443: connect: connection refused" 2024-04-12T22:02:33.743 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:33 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: 2024-04-12T22:02:33.328+0000 7fe6f24ab200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-12T22:02:33.743 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:33 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: 2024-04-12T22:02:33.422+0000 7fe6f24ab200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-12T22:02:34.392 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:34 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: 2024-04-12T22:02:34.059+0000 7fe6f24ab200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-12T22:02:34.392 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:34 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: 2024-04-12T22:02:34.151+0000 7fe6f24ab200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-12T22:02:34.392 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:34 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: 2024-04-12T22:02:34.292+0000 7fe6f24ab200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-12T22:02:34.392 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:34 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: 2024-04-12T22:02:34.390+0000 7fe6f24ab200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-12T22:02:34.410 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:02:34 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:02:34.094+0000 7fe4598af200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-12T22:02:34.410 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:02:34 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:02:34.188+0000 7fe4598af200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-12T22:02:34.410 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:02:34 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:02:34.331+0000 7fe4598af200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-12T22:02:34.738 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:34 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: 2024-04-12T22:02:34.473+0000 7fe6f24ab200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-12T22:02:34.738 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:34 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: 2024-04-12T22:02:34.736+0000 7fe6f24ab200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-12T22:02:34.764 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:02:34 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:02:34.427+0000 7fe4598af200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-12T22:02:34.765 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:02:34 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:02:34.511+0000 7fe4598af200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-12T22:02:34.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:34 smithi114 ceph-mon[25759]: mgrmap e33: y(active, since 40s), standbys: x 2024-04-12T22:02:34.994 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:34 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: 2024-04-12T22:02:34.846+0000 7fe6f24ab200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-12T22:02:34.994 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:34 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: 2024-04-12T22:02:34.931+0000 7fe6f24ab200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-12T22:02:35.071 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:34 smithi097 ceph-mon[64853]: mgrmap e33: y(active, since 40s), standbys: x 2024-04-12T22:02:35.072 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:34 smithi097 ceph-mon[29620]: mgrmap e33: y(active, since 40s), standbys: x 2024-04-12T22:02:35.072 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:02:34 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:02:34.778+0000 7fe4598af200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-12T22:02:35.072 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:02:34 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:02:34.888+0000 7fe4598af200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-12T22:02:35.072 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:02:34 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:02:34.974+0000 7fe4598af200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-12T22:02:35.072 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:02:35 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:02:35.070+0000 7fe4598af200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-12T22:02:35.410 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:02:35 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:02:35.249+0000 7fe4598af200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-12T22:02:35.493 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:35 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: 2024-04-12T22:02:35.028+0000 7fe6f24ab200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-12T22:02:35.493 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:35 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: 2024-04-12T22:02:35.204+0000 7fe6f24ab200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-12T22:02:36.493 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:36 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: 2024-04-12T22:02:36.159+0000 7fe6f24ab200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-12T22:02:36.493 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:36 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: 2024-04-12T22:02:36.291+0000 7fe6f24ab200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-12T22:02:36.660 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:02:36 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:02:36.207+0000 7fe4598af200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-12T22:02:36.660 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:02:36 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:02:36.340+0000 7fe4598af200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-12T22:02:36.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:36 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: 2024-04-12T22:02:36.625+0000 7fe6f24ab200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-12T22:02:36.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:36 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: 2024-04-12T22:02:36.716+0000 7fe6f24ab200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-12T22:02:36.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:36 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: 2024-04-12T22:02:36.914+0000 7fe6f24ab200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-12T22:02:36.969 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:02:36 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:02:36.677+0000 7fe4598af200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-12T22:02:36.969 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:02:36 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:02:36.769+0000 7fe4598af200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-12T22:02:36.969 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:02:36 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:02:36.968+0000 7fe4598af200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-12T22:02:37.243 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:37 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: 2024-04-12T22:02:37.102+0000 7fe6f24ab200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-12T22:02:37.410 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:02:37 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:02:37.157+0000 7fe4598af200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-12T22:02:37.660 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:37 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: 2024-04-12T22:02:37.390+0000 7fe6f24ab200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-12T22:02:37.711 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:02:37 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:02:37.449+0000 7fe4598af200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-12T22:02:37.711 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:02:37 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:02:37.710+0000 7fe4598af200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-12T22:02:37.993 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:37 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: 2024-04-12T22:02:37.658+0000 7fe6f24ab200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-12T22:02:37.993 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:37 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: 2024-04-12T22:02:37.895+0000 7fe6f24ab200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-12T22:02:38.160 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:02:37 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:02:37.951+0000 7fe4598af200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-12T22:02:38.993 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:38 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: 2024-04-12T22:02:38.706+0000 7fe6f24ab200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-12T22:02:38.993 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:02:38 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:02:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"smithi097:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"172.21.15.114:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-12T22:02:39.160 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:02:38 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:02:38.764+0000 7fe4598af200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-12T22:02:39.606 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:39 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: 2024-04-12T22:02:39.280+0000 7fe6f24ab200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-12T22:02:39.606 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:39 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: 2024-04-12T22:02:39.382+0000 7fe6f24ab200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-12T22:02:39.660 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:02:39 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:02:39.339+0000 7fe4598af200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-12T22:02:39.660 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:02:39 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:02:39.442+0000 7fe4598af200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-12T22:02:39.899 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:39 smithi114 ceph-mon[25759]: Standby manager daemon x restarted 2024-04-12T22:02:39.899 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:39 smithi114 ceph-mon[25759]: Standby manager daemon x started 2024-04-12T22:02:39.899 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:39 smithi114 ceph-mon[25759]: from='mgr.? 172.21.15.114:0/3248806247' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-12T22:02:39.899 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:39 smithi114 ceph-mon[25759]: from='mgr.? 172.21.15.114:0/3248806247' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-12T22:02:39.899 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:39 smithi114 ceph-mon[25759]: from='mgr.? 172.21.15.114:0/3248806247' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-12T22:02:39.899 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:39 smithi114 ceph-mon[25759]: from='mgr.? 172.21.15.114:0/3248806247' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-12T22:02:39.899 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:39 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: 2024-04-12T22:02:39.604+0000 7fe6f24ab200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-12T22:02:39.900 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:39 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: [12/Apr/2024:22:02:39] ENGINE Bus STARTING 2024-04-12T22:02:39.900 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:39 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: CherryPy Checker: 2024-04-12T22:02:39.900 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:39 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: The Application mounted at '' has an empty config. 2024-04-12T22:02:39.900 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:39 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: 2024-04-12T22:02:39.900 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:39 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: [12/Apr/2024:22:02:39] ENGINE Serving on http://:::9283 2024-04-12T22:02:39.900 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:02:39 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x[54755]: [12/Apr/2024:22:02:39] ENGINE Bus STARTED 2024-04-12T22:02:40.160 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:02:39 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[59315]: level=info ts=2024-04-12T22:02:39.791Z caller=notify.go:735 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify success" attempts=6 2024-04-12T22:02:40.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:39 smithi097 ceph-mon[64853]: Standby manager daemon x restarted 2024-04-12T22:02:40.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:39 smithi097 ceph-mon[64853]: Standby manager daemon x started 2024-04-12T22:02:40.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:39 smithi097 ceph-mon[64853]: from='mgr.? 172.21.15.114:0/3248806247' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-12T22:02:40.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:39 smithi097 ceph-mon[64853]: from='mgr.? 172.21.15.114:0/3248806247' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-12T22:02:40.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:39 smithi097 ceph-mon[64853]: from='mgr.? 172.21.15.114:0/3248806247' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-12T22:02:40.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:39 smithi097 ceph-mon[64853]: from='mgr.? 172.21.15.114:0/3248806247' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-12T22:02:40.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:39 smithi097 ceph-mon[29620]: Standby manager daemon x restarted 2024-04-12T22:02:40.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:39 smithi097 ceph-mon[29620]: Standby manager daemon x started 2024-04-12T22:02:40.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:39 smithi097 ceph-mon[29620]: from='mgr.? 172.21.15.114:0/3248806247' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-12T22:02:40.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:39 smithi097 ceph-mon[29620]: from='mgr.? 172.21.15.114:0/3248806247' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-12T22:02:40.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:39 smithi097 ceph-mon[29620]: from='mgr.? 172.21.15.114:0/3248806247' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-12T22:02:40.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:39 smithi097 ceph-mon[29620]: from='mgr.? 172.21.15.114:0/3248806247' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-12T22:02:40.162 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:02:39 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:02:39.668+0000 7fe4598af200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-12T22:02:40.162 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:02:39 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: [12/Apr/2024:22:02:39] ENGINE Bus STARTING 2024-04-12T22:02:40.162 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:02:39 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: CherryPy Checker: 2024-04-12T22:02:40.162 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:02:39 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: The Application mounted at '' has an empty config. 2024-04-12T22:02:40.162 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:02:39 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:02:40.162 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:02:39 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: [12/Apr/2024:22:02:39] ENGINE Serving on http://:::9283 2024-04-12T22:02:40.162 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:02:39 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: [12/Apr/2024:22:02:39] ENGINE Bus STARTED 2024-04-12T22:02:40.717 INFO:teuthology.orchestra.run.smithi097.stdout:true 2024-04-12T22:02:40.973 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:40 smithi114 ceph-mon[25759]: mgrmap e34: y(active, since 46s), standbys: x 2024-04-12T22:02:40.973 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:40 smithi114 ceph-mon[25759]: Active manager daemon y restarted 2024-04-12T22:02:40.974 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:40 smithi114 ceph-mon[25759]: Activating manager daemon y 2024-04-12T22:02:40.974 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:40 smithi114 ceph-mon[25759]: osdmap e106: 8 total, 8 up, 8 in 2024-04-12T22:02:40.974 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:40 smithi114 ceph-mon[25759]: mgrmap e35: y(active, starting, since 0.00893057s), standbys: x 2024-04-12T22:02:40.974 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:40 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-12T22:02:40.974 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:40 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T22:02:40.974 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:40 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T22:02:40.974 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:40 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-12T22:02:40.974 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:40 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-12T22:02:40.974 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:40 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T22:02:40.974 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:40 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T22:02:40.974 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:40 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T22:02:40.974 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:40 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T22:02:40.975 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:40 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T22:02:40.975 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:40 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T22:02:40.975 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:40 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-12T22:02:40.975 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:40 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-12T22:02:40.975 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:40 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-12T22:02:40.975 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:40 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-12T22:02:40.975 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:40 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-12T22:02:40.975 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:40 smithi114 ceph-mon[25759]: Manager daemon y is now available 2024-04-12T22:02:40.975 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:40 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:02:40.975 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:40 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T22:02:40.975 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:40 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T22:02:41.056 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:02:40 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:02:40.690+0000 7fe40fa5a700 -1 mgr.server handle_report got status from non-daemon mon.a 2024-04-12T22:02:41.057 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:40 smithi097 ceph-mon[64853]: mgrmap e34: y(active, since 46s), standbys: x 2024-04-12T22:02:41.057 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:40 smithi097 ceph-mon[64853]: Active manager daemon y restarted 2024-04-12T22:02:41.057 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:40 smithi097 ceph-mon[64853]: Activating manager daemon y 2024-04-12T22:02:41.057 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:40 smithi097 ceph-mon[64853]: osdmap e106: 8 total, 8 up, 8 in 2024-04-12T22:02:41.057 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:40 smithi097 ceph-mon[64853]: mgrmap e35: y(active, starting, since 0.00893057s), standbys: x 2024-04-12T22:02:41.057 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:40 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-12T22:02:41.057 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:40 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T22:02:41.058 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:40 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T22:02:41.058 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:40 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-12T22:02:41.058 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:40 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-12T22:02:41.058 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:40 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T22:02:41.058 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:40 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T22:02:41.058 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:40 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T22:02:41.058 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:40 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T22:02:41.058 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:40 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T22:02:41.058 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:40 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T22:02:41.058 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:40 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-12T22:02:41.058 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:40 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-12T22:02:41.058 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:40 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-12T22:02:41.058 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:40 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-12T22:02:41.059 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:40 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-12T22:02:41.059 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:40 smithi097 ceph-mon[64853]: Manager daemon y is now available 2024-04-12T22:02:41.059 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:40 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:02:41.059 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:40 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T22:02:41.059 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:40 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T22:02:41.059 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:40 smithi097 ceph-mon[29620]: mgrmap e34: y(active, since 46s), standbys: x 2024-04-12T22:02:41.059 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:40 smithi097 ceph-mon[29620]: Active manager daemon y restarted 2024-04-12T22:02:41.059 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:40 smithi097 ceph-mon[29620]: Activating manager daemon y 2024-04-12T22:02:41.059 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:40 smithi097 ceph-mon[29620]: osdmap e106: 8 total, 8 up, 8 in 2024-04-12T22:02:41.059 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:40 smithi097 ceph-mon[29620]: mgrmap e35: y(active, starting, since 0.00893057s), standbys: x 2024-04-12T22:02:41.059 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:40 smithi097 ceph-mon[29620]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-12T22:02:41.059 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:40 smithi097 ceph-mon[29620]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T22:02:41.059 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:40 smithi097 ceph-mon[29620]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T22:02:41.059 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:40 smithi097 ceph-mon[29620]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-12T22:02:41.060 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:40 smithi097 ceph-mon[29620]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-12T22:02:41.060 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:40 smithi097 ceph-mon[29620]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T22:02:41.060 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:40 smithi097 ceph-mon[29620]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T22:02:41.060 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:40 smithi097 ceph-mon[29620]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T22:02:41.060 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:40 smithi097 ceph-mon[29620]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T22:02:41.060 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:40 smithi097 ceph-mon[29620]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T22:02:41.060 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:40 smithi097 ceph-mon[29620]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T22:02:41.060 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:40 smithi097 ceph-mon[29620]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-12T22:02:41.060 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:40 smithi097 ceph-mon[29620]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-12T22:02:41.060 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:40 smithi097 ceph-mon[29620]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-12T22:02:41.060 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:40 smithi097 ceph-mon[29620]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-12T22:02:41.060 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:40 smithi097 ceph-mon[29620]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-12T22:02:41.060 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:40 smithi097 ceph-mon[29620]: Manager daemon y is now available 2024-04-12T22:02:41.060 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:40 smithi097 ceph-mon[29620]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:02:41.060 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:40 smithi097 ceph-mon[29620]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-12T22:02:41.060 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:40 smithi097 ceph-mon[29620]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-12T22:02:41.379 INFO:teuthology.orchestra.run.smithi097.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T22:02:41.379 INFO:teuthology.orchestra.run.smithi097.stdout:alertmanager.a smithi097 *:9093,9094 running (97s) 44s ago 6m 13.6M - 0.23.0 ba2b418f427c 7b4c491fe0a0 2024-04-12T22:02:41.379 INFO:teuthology.orchestra.run.smithi097.stdout:grafana.a smithi114 *:3000 unknown 22s ago 6m - - 2024-04-12T22:02:41.379 INFO:teuthology.orchestra.run.smithi097.stdout:iscsi.foo.smithi097.xndkxy smithi097 unknown 44s ago 5m - - 2024-04-12T22:02:41.379 INFO:teuthology.orchestra.run.smithi097.stdout:mgr.x smithi114 *:8443,9283,8765 running (27s) 22s ago 11m 68.7M - 19.0.0-2053-gb277feb2 668895944f0e b8f8df3bf928 2024-04-12T22:02:41.379 INFO:teuthology.orchestra.run.smithi097.stdout:mgr.y smithi097 *:8443,9283,8765 running (63s) 44s ago 12m 475M - 19.0.0-2053-gb277feb2 668895944f0e 488170dac11f 2024-04-12T22:02:41.379 INFO:teuthology.orchestra.run.smithi097.stdout:mon.a smithi097 running (12m) 44s ago 12m 52.1M 2048M 17.2.0 e1d6a67b021e 6e618120f594 2024-04-12T22:02:41.379 INFO:teuthology.orchestra.run.smithi097.stdout:mon.b smithi114 running (11m) 22s ago 11m 49.7M 2048M 17.2.0 e1d6a67b021e 47a08e70f856 2024-04-12T22:02:41.379 INFO:teuthology.orchestra.run.smithi097.stdout:mon.c smithi097 running (11m) 44s ago 11m 47.5M 2048M 17.2.0 e1d6a67b021e 679b33a29a0c 2024-04-12T22:02:41.379 INFO:teuthology.orchestra.run.smithi097.stdout:node-exporter.a smithi097 *:9100 running (107s) 44s ago 7m 11.3M - 1.3.1 1dbe0e931976 5edff9053b37 2024-04-12T22:02:41.379 INFO:teuthology.orchestra.run.smithi097.stdout:node-exporter.b smithi114 *:9100 running (79s) 22s ago 7m 8476k - 1.3.1 1dbe0e931976 904afc45dc2f 2024-04-12T22:02:41.379 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0 smithi097 running (10m) 44s ago 10m 54.0M 2736M 17.2.0 e1d6a67b021e abac1d8884fa 2024-04-12T22:02:41.379 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1 smithi097 running (10m) 44s ago 10m 52.7M 2736M 17.2.0 e1d6a67b021e 39192562d598 2024-04-12T22:02:41.380 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2 smithi097 running (9m) 44s ago 9m 50.7M 2736M 17.2.0 e1d6a67b021e 89daf8e339c3 2024-04-12T22:02:41.380 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3 smithi097 running (9m) 44s ago 9m 51.1M 2736M 17.2.0 e1d6a67b021e 4e9a0b3862d9 2024-04-12T22:02:41.380 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4 smithi114 running (9m) 22s ago 9m 53.0M 2992M 17.2.0 e1d6a67b021e fb42c0e8e575 2024-04-12T22:02:41.380 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5 smithi114 running (8m) 22s ago 8m 49.7M 2992M 17.2.0 e1d6a67b021e 3acd4a3b61e5 2024-04-12T22:02:41.380 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6 smithi114 running (8m) 22s ago 8m 49.0M 2992M 17.2.0 e1d6a67b021e 5969baf67f41 2024-04-12T22:02:41.380 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7 smithi114 running (8m) 22s ago 8m 51.2M 2992M 17.2.0 e1d6a67b021e f68340fecf2c 2024-04-12T22:02:41.380 INFO:teuthology.orchestra.run.smithi097.stdout:prometheus.a smithi114 *:9095 running (33s) 22s ago 7m 43.0M - 2.33.4 514e6a882f6e 0ad5437ed63f 2024-04-12T22:02:41.380 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.foo.smithi097.sggtkh smithi097 *:8000 running (5m) 44s ago 5m 71.0M - 17.2.0 e1d6a67b021e a39f9b2d8509 2024-04-12T22:02:41.380 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.foo.smithi114.ulfbyo smithi114 *:8000 running (5m) 22s ago 5m 72.0M - 17.2.0 e1d6a67b021e df79a48bdd59 2024-04-12T22:02:41.380 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.smpl.smithi097.exxkgl smithi097 *:80 running (5m) 44s ago 5m 71.2M - 17.2.0 e1d6a67b021e a036c58a0099 2024-04-12T22:02:41.380 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.smpl.smithi114.lrgvph smithi114 *:80 running (5m) 22s ago 5m 73.6M - 17.2.0 e1d6a67b021e 031a5fc24237 2024-04-12T22:02:41.803 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-12T22:02:41.803 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": { 2024-04-12T22:02:41.803 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-04-12T22:02:41.803 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 1 2024-04-12T22:02:41.804 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:02:41.804 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": { 2024-04-12T22:02:41.804 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 2 2024-04-12T22:02:41.804 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:02:41.804 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": { 2024-04-12T22:02:41.804 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-12T22:02:41.804 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:02:41.804 INFO:teuthology.orchestra.run.smithi097.stdout: "mds": {}, 2024-04-12T22:02:41.804 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw": { 2024-04-12T22:02:41.804 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-12T22:02:41.804 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:02:41.804 INFO:teuthology.orchestra.run.smithi097.stdout: "overall": { 2024-04-12T22:02:41.804 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14, 2024-04-12T22:02:41.804 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 3 2024-04-12T22:02:41.805 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-12T22:02:41.805 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-12T22:02:41.826 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:41 smithi114 ceph-mon[25759]: mgrmap e36: y(active, since 1.01405s), standbys: x 2024-04-12T22:02:41.827 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:41 smithi114 ceph-mon[25759]: [12/Apr/2024:22:02:40] ENGINE Bus STARTING 2024-04-12T22:02:41.827 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:41 smithi114 ceph-mon[25759]: [12/Apr/2024:22:02:40] ENGINE Serving on http://172.21.15.97:8765 2024-04-12T22:02:41.960 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:41 smithi097 ceph-mon[64853]: mgrmap e36: y(active, since 1.01405s), standbys: x 2024-04-12T22:02:41.961 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:41 smithi097 ceph-mon[64853]: [12/Apr/2024:22:02:40] ENGINE Bus STARTING 2024-04-12T22:02:41.961 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:41 smithi097 ceph-mon[64853]: [12/Apr/2024:22:02:40] ENGINE Serving on http://172.21.15.97:8765 2024-04-12T22:02:41.961 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:41 smithi097 ceph-mon[29620]: mgrmap e36: y(active, since 1.01405s), standbys: x 2024-04-12T22:02:41.961 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:41 smithi097 ceph-mon[29620]: [12/Apr/2024:22:02:40] ENGINE Bus STARTING 2024-04-12T22:02:41.961 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:41 smithi097 ceph-mon[29620]: [12/Apr/2024:22:02:40] ENGINE Serving on http://172.21.15.97:8765 2024-04-12T22:02:42.141 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-12T22:02:42.141 INFO:teuthology.orchestra.run.smithi097.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a", 2024-04-12T22:02:42.141 INFO:teuthology.orchestra.run.smithi097.stdout: "in_progress": true, 2024-04-12T22:02:42.141 INFO:teuthology.orchestra.run.smithi097.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T22:02:42.141 INFO:teuthology.orchestra.run.smithi097.stdout: "services_complete": [ 2024-04-12T22:02:42.141 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr" 2024-04-12T22:02:42.141 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-04-12T22:02:42.141 INFO:teuthology.orchestra.run.smithi097.stdout: "progress": "2/23 daemons upgraded", 2024-04-12T22:02:42.142 INFO:teuthology.orchestra.run.smithi097.stdout: "message": "", 2024-04-12T22:02:42.142 INFO:teuthology.orchestra.run.smithi097.stdout: "is_paused": false 2024-04-12T22:02:42.142 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-12T22:02:42.572 INFO:teuthology.orchestra.run.smithi097.stdout:HEALTH_OK 2024-04-12T22:02:42.743 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:42 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@grafana.a.service: Scheduled restart job, restart counter is at 6. 2024-04-12T22:02:42.743 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:42 smithi114 systemd[1]: Stopped Ceph grafana.a for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:02:42.743 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:42 smithi114 systemd[1]: Starting Ceph grafana.a for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:02:42.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:42 smithi114 ceph-mon[25759]: [12/Apr/2024:22:02:40] ENGINE Serving on https://172.21.15.97:7150 2024-04-12T22:02:42.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:42 smithi114 ceph-mon[25759]: [12/Apr/2024:22:02:40] ENGINE Bus STARTED 2024-04-12T22:02:42.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:42 smithi114 ceph-mon[25759]: from='client.25366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:02:42.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:42 smithi114 ceph-mon[25759]: from='client.25372 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:02:42.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:42 smithi114 ceph-mon[25759]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 2 failed cephadm daemon(s)) 2024-04-12T22:02:42.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:42 smithi114 ceph-mon[25759]: Cluster is now healthy 2024-04-12T22:02:42.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:42 smithi114 ceph-mon[25759]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:02:42.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:42 smithi114 ceph-mon[25759]: mgrmap e37: y(active, since 2s), standbys: x 2024-04-12T22:02:42.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:42 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/4216623317' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:02:42.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:42 smithi114 ceph-mon[25759]: from='client.? 172.21.15.97:0/387714461' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T22:02:42.848 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:42 smithi097 ceph-mon[64853]: [12/Apr/2024:22:02:40] ENGINE Serving on https://172.21.15.97:7150 2024-04-12T22:02:42.848 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:42 smithi097 ceph-mon[64853]: [12/Apr/2024:22:02:40] ENGINE Bus STARTED 2024-04-12T22:02:42.848 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:42 smithi097 ceph-mon[64853]: from='client.25366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:02:42.848 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:42 smithi097 ceph-mon[64853]: from='client.25372 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:02:42.848 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:42 smithi097 ceph-mon[64853]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 2 failed cephadm daemon(s)) 2024-04-12T22:02:42.848 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:42 smithi097 ceph-mon[64853]: Cluster is now healthy 2024-04-12T22:02:42.848 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:42 smithi097 ceph-mon[64853]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:02:42.848 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:42 smithi097 ceph-mon[64853]: mgrmap e37: y(active, since 2s), standbys: x 2024-04-12T22:02:42.848 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:42 smithi097 ceph-mon[64853]: from='client.? 172.21.15.97:0/4216623317' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:02:42.848 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:42 smithi097 ceph-mon[64853]: from='client.? 172.21.15.97:0/387714461' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T22:02:42.849 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:42 smithi097 ceph-mon[29620]: [12/Apr/2024:22:02:40] ENGINE Serving on https://172.21.15.97:7150 2024-04-12T22:02:42.849 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:42 smithi097 ceph-mon[29620]: [12/Apr/2024:22:02:40] ENGINE Bus STARTED 2024-04-12T22:02:42.849 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:42 smithi097 ceph-mon[29620]: from='client.25366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:02:42.849 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:42 smithi097 ceph-mon[29620]: from='client.25372 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:02:42.849 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:42 smithi097 ceph-mon[29620]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 2 failed cephadm daemon(s)) 2024-04-12T22:02:42.849 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:42 smithi097 ceph-mon[29620]: Cluster is now healthy 2024-04-12T22:02:42.849 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:42 smithi097 ceph-mon[29620]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:02:42.849 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:42 smithi097 ceph-mon[29620]: mgrmap e37: y(active, since 2s), standbys: x 2024-04-12T22:02:42.849 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:42 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/4216623317' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:02:42.849 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:42 smithi097 ceph-mon[29620]: from='client.? 172.21.15.97:0/387714461' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T22:02:43.660 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:02:43 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[59315]: level=info ts=2024-04-12T22:02:43.508Z caller=notify.go:735 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify success" attempts=7 2024-04-12T22:02:43.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:43 smithi114 ceph-mon[25759]: from='client.34158 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:02:43.993 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:43 smithi114 podman[56671]: 2024-04-12 22:02:43.743345034 +0000 UTC m=+0.109881288 container create a4fb97aa55786765d757fe52f3a6a64cb2da6f288e92b101bcdc7fd6c7e72e03 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.component=ubi8-container, summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, distribution-scope=public, description=Ceph Grafana Container, name=ubi8, io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , version=8.5, io.buildah.version=1.24.2, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=236.1648460182, io.openshift.tags=base rhel8, build-date=2022-03-28T10:36:18.413762, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vendor=Red Hat, Inc., vcs-type=git) 2024-04-12T22:02:43.993 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:43 smithi114 podman[56671]: 2024-04-12 22:02:43.652611966 +0000 UTC m=+0.019148221 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-04-12T22:02:44.028 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:43 smithi097 ceph-mon[64853]: from='client.34158 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:02:44.029 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:43 smithi097 ceph-mon[29620]: from='client.34158 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:02:44.493 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:44 smithi114 podman[56671]: 2024-04-12 22:02:44.037458505 +0000 UTC m=+0.403994759 container init a4fb97aa55786765d757fe52f3a6a64cb2da6f288e92b101bcdc7fd6c7e72e03 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, summary=Grafana Container configured for Ceph mgr/dashboard integration, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-03-28T10:36:18.413762, distribution-scope=public, io.openshift.tags=base rhel8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=ubi8, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.component=ubi8-container, version=8.5, release=236.1648460182, description=Ceph Grafana Container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.24.2) 2024-04-12T22:02:44.493 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:44 smithi114 podman[56671]: 2024-04-12 22:02:44.04031071 +0000 UTC m=+0.406846968 container start a4fb97aa55786765d757fe52f3a6a64cb2da6f288e92b101bcdc7fd6c7e72e03 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, build-date=2022-03-28T10:36:18.413762, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.24.2, maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, name=ubi8, distribution-scope=public, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-type=git, description=Ceph Grafana Container, version=8.5, release=236.1648460182, io.openshift.tags=base rhel8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., io.openshift.expose-services=, com.redhat.component=ubi8-container) 2024-04-12T22:02:44.493 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:44 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[56699]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-12T22:02:44.494 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:44 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[56699]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-12T22:02:44.494 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:44 smithi114 bash[56671]: a4fb97aa55786765d757fe52f3a6a64cb2da6f288e92b101bcdc7fd6c7e72e03 2024-04-12T22:02:44.494 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:44 smithi114 systemd[1]: Started Ceph grafana.a for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:02:44.494 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:44 smithi114 podman[56729]: 2024-04-12 22:02:44.149941631 +0000 UTC m=+0.072299090 container died a4fb97aa55786765d757fe52f3a6a64cb2da6f288e92b101bcdc7fd6c7e72e03 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, version=8.5, com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, name=ubi8, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=Red Hat Universal Base Image 8, io.buildah.version=1.24.2, maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.expose-services=, release=236.1648460182, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.openshift.tags=base rhel8, build-date=2022-03-28T10:36:18.413762, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., vcs-type=git, distribution-scope=public, summary=Grafana Container configured for Ceph mgr/dashboard integration, description=Ceph Grafana Container) 2024-04-12T22:02:44.993 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:44 smithi114 podman[56729]: 2024-04-12 22:02:44.699401128 +0000 UTC m=+0.621758673 container remove a4fb97aa55786765d757fe52f3a6a64cb2da6f288e92b101bcdc7fd6c7e72e03 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration, name=ubi8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8.5, io.openshift.expose-services=, vendor=Red Hat, Inc., description=Ceph Grafana Container, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, architecture=x86_64, io.buildah.version=1.24.2, maintainer=Paul Cuzner , com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, release=236.1648460182, build-date=2022-03-28T10:36:18.413762, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.component=ubi8-container, distribution-scope=public) 2024-04-12T22:02:44.993 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:44 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-04-12T22:02:44.993 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:44 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-04-12T22:02:44.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:44 smithi114 ceph-mon[25759]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:02:44.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:44 smithi114 ceph-mon[25759]: mgrmap e38: y(active, since 4s), standbys: x 2024-04-12T22:02:44.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:44 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:02:44.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:44 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:02:45.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:44 smithi097 ceph-mon[64853]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:02:45.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:44 smithi097 ceph-mon[64853]: mgrmap e38: y(active, since 4s), standbys: x 2024-04-12T22:02:45.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:44 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:02:45.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:44 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:02:45.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:44 smithi097 ceph-mon[29620]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:02:45.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:44 smithi097 ceph-mon[29620]: mgrmap e38: y(active, since 4s), standbys: x 2024-04-12T22:02:45.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:44 smithi097 ceph-mon[29620]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:02:45.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:44 smithi097 ceph-mon[29620]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:02:46.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:45 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:02:46.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:45 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:02:46.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:45 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:02:46.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:45 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:02:46.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:45 smithi097 ceph-mon[64853]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:02:46.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:45 smithi097 ceph-mon[29620]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:02:46.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:45 smithi097 ceph-mon[29620]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:02:46.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:45 smithi097 ceph-mon[29620]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:02:46.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:45 smithi097 ceph-mon[29620]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:02:46.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:45 smithi097 ceph-mon[29620]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:02:46.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:45 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:02:46.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:45 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:02:46.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:45 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:02:46.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:45 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:02:46.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:45 smithi114 ceph-mon[25759]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:02:47.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:46 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:02:47.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:46 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:02:47.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:46 smithi097 ceph-mon[29620]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:02:47.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:46 smithi097 ceph-mon[29620]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:02:47.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:46 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:02:47.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:46 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:02:48.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:47 smithi097 ceph-mon[64853]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-12T22:02:48.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:47 smithi097 ceph-mon[29620]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-12T22:02:48.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:47 smithi114 ceph-mon[25759]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-12T22:02:49.660 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:02:49 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:02:49] "GET /metrics HTTP/1.1" 200 35449 "" "Prometheus/2.33.4" 2024-04-12T22:02:50.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:50 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:02:50.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:50 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:02:50.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:50 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:02:50.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:50 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:02:50.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:50 smithi097 ceph-mon[64853]: Updating smithi097:/etc/ceph/ceph.conf 2024-04-12T22:02:50.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:50 smithi097 ceph-mon[64853]: Updating smithi114:/etc/ceph/ceph.conf 2024-04-12T22:02:50.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:50 smithi097 ceph-mon[64853]: Updating smithi114:/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/config/ceph.conf 2024-04-12T22:02:50.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:50 smithi097 ceph-mon[64853]: Updating smithi097:/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/config/ceph.conf 2024-04-12T22:02:50.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:50 smithi097 ceph-mon[64853]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-12T22:02:50.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:50 smithi097 ceph-mon[64853]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-04-12T22:02:50.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:50 smithi097 ceph-mon[64853]: Updating smithi097:/etc/ceph/ceph.client.admin.keyring 2024-04-12T22:02:50.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:50 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:02:50.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:50 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:02:50.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:50 smithi097 ceph-mon[29620]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:02:50.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:50 smithi097 ceph-mon[29620]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:02:50.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:50 smithi097 ceph-mon[29620]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:02:50.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:50 smithi097 ceph-mon[29620]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:02:50.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:50 smithi097 ceph-mon[29620]: Updating smithi097:/etc/ceph/ceph.conf 2024-04-12T22:02:50.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:50 smithi097 ceph-mon[29620]: Updating smithi114:/etc/ceph/ceph.conf 2024-04-12T22:02:50.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:50 smithi097 ceph-mon[29620]: Updating smithi114:/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/config/ceph.conf 2024-04-12T22:02:50.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:50 smithi097 ceph-mon[29620]: Updating smithi097:/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/config/ceph.conf 2024-04-12T22:02:50.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:50 smithi097 ceph-mon[29620]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-12T22:02:50.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:50 smithi097 ceph-mon[29620]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-04-12T22:02:50.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:50 smithi097 ceph-mon[29620]: Updating smithi097:/etc/ceph/ceph.client.admin.keyring 2024-04-12T22:02:50.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:50 smithi097 ceph-mon[29620]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:02:50.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:50 smithi097 ceph-mon[29620]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:02:50.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:50 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:02:50.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:50 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:02:50.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:50 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:02:50.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:50 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:02:50.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:50 smithi114 ceph-mon[25759]: Updating smithi097:/etc/ceph/ceph.conf 2024-04-12T22:02:50.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:50 smithi114 ceph-mon[25759]: Updating smithi114:/etc/ceph/ceph.conf 2024-04-12T22:02:50.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:50 smithi114 ceph-mon[25759]: Updating smithi114:/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/config/ceph.conf 2024-04-12T22:02:50.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:50 smithi114 ceph-mon[25759]: Updating smithi097:/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/config/ceph.conf 2024-04-12T22:02:50.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:50 smithi114 ceph-mon[25759]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-12T22:02:50.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:50 smithi114 ceph-mon[25759]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-04-12T22:02:50.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:50 smithi114 ceph-mon[25759]: Updating smithi097:/etc/ceph/ceph.client.admin.keyring 2024-04-12T22:02:50.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:50 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:02:50.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:50 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:02:51.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:51 smithi097 ceph-mon[64853]: Updating smithi114:/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/config/ceph.client.admin.keyring 2024-04-12T22:02:51.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:51 smithi097 ceph-mon[64853]: Updating smithi097:/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/config/ceph.client.admin.keyring 2024-04-12T22:02:51.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:51 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:02:51.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:51 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:02:51.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:51 smithi097 ceph-mon[64853]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-12T22:02:51.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:51 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:02:51.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:51 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:02:51.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:51 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:02:51.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:51 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-12T22:02:51.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:51 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-04-12T22:02:51.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:51 smithi097 ceph-mon[64853]: Upgrade: It appears safe to stop mon.c 2024-04-12T22:02:51.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:51 smithi097 ceph-mon[64853]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:02:51.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:51 smithi097 ceph-mon[29620]: Updating smithi114:/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/config/ceph.client.admin.keyring 2024-04-12T22:02:51.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:51 smithi097 ceph-mon[29620]: Updating smithi097:/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/config/ceph.client.admin.keyring 2024-04-12T22:02:51.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:51 smithi097 ceph-mon[29620]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:02:51.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:51 smithi097 ceph-mon[29620]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:02:51.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:51 smithi097 ceph-mon[29620]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-12T22:02:51.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:51 smithi097 ceph-mon[29620]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:02:51.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:51 smithi097 ceph-mon[29620]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:02:51.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:51 smithi097 ceph-mon[29620]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:02:51.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:51 smithi097 ceph-mon[29620]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-12T22:02:51.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:51 smithi097 ceph-mon[29620]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-04-12T22:02:51.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:51 smithi097 ceph-mon[29620]: Upgrade: It appears safe to stop mon.c 2024-04-12T22:02:51.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:51 smithi097 ceph-mon[29620]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:02:51.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:51 smithi114 ceph-mon[25759]: Updating smithi114:/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/config/ceph.client.admin.keyring 2024-04-12T22:02:51.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:51 smithi114 ceph-mon[25759]: Updating smithi097:/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/config/ceph.client.admin.keyring 2024-04-12T22:02:51.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:51 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:02:51.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:51 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:02:51.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:51 smithi114 ceph-mon[25759]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-12T22:02:51.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:51 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:02:51.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:51 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:02:51.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:51 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:02:51.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:51 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-12T22:02:51.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:51 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-04-12T22:02:51.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:51 smithi114 ceph-mon[25759]: Upgrade: It appears safe to stop mon.c 2024-04-12T22:02:51.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:51 smithi114 ceph-mon[25759]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:02:52.623 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:52 smithi097 ceph-mon[64853]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-12T22:02:52.623 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:52 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:02:52.623 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:52 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T22:02:52.623 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:52 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T22:02:52.623 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:52 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:02:52.624 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:52 smithi097 ceph-mon[29620]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-12T22:02:52.624 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:52 smithi097 ceph-mon[29620]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:02:52.624 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:52 smithi097 ceph-mon[29620]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T22:02:52.624 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:52 smithi097 ceph-mon[29620]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T22:02:52.624 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:52 smithi097 ceph-mon[29620]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:02:52.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:52 smithi114 ceph-mon[25759]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-12T22:02:52.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:52 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:02:52.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:52 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T22:02:52.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:52 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T22:02:52.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:52 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:02:53.363 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:53 smithi097 ceph-mon[64853]: Upgrade: Updating mon.c 2024-04-12T22:02:53.363 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:53 smithi097 ceph-mon[64853]: Deploying daemon mon.c on smithi097 2024-04-12T22:02:53.363 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:53 smithi097 ceph-mon[64853]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-12T22:02:53.364 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:53 smithi097 ceph-mon[29620]: Upgrade: Updating mon.c 2024-04-12T22:02:53.364 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:53 smithi097 ceph-mon[29620]: Deploying daemon mon.c on smithi097 2024-04-12T22:02:53.364 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:53 smithi097 ceph-mon[29620]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-12T22:02:53.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:53 smithi114 ceph-mon[25759]: Upgrade: Updating mon.c 2024-04-12T22:02:53.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:53 smithi114 ceph-mon[25759]: Deploying daemon mon.c on smithi097 2024-04-12T22:02:53.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:53 smithi114 ceph-mon[25759]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-12T22:02:53.904 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:53 smithi097 systemd[1]: Stopping Ceph mon.c for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:02:53.904 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mon-c[29616]: 2024-04-12T22:02:53.900+0000 7ffb82b51700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-12T22:02:54.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:53 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mon-c[29616]: 2024-04-12T22:02:53.900+0000 7ffb82b51700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-12T22:02:54.502 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:54 smithi097 podman[67240]: 2024-04-12 22:02:54.166133156 +0000 UTC m=+0.372156279 container died 679b33a29a0cc1a76fb83b8403c75c795f938108068da55843d41c2882859362 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mon-c, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , distribution-scope=public, vcs-type=git, ceph=True, version=8, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, release=754, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8) 2024-04-12T22:02:54.818 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:54 smithi097 podman[67240]: 2024-04-12 22:02:54.600842201 +0000 UTC m=+0.806865337 container cleanup 679b33a29a0cc1a76fb83b8403c75c795f938108068da55843d41c2882859362 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mon-c, GIT_CLEAN=True, maintainer=Guillaume Abrioux , architecture=x86_64, io.openshift.tags=base centos centos-stream, ceph=True, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, GIT_BRANCH=HEAD, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-04-12T22:02:54.819 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:54 smithi097 bash[67240]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mon-c 2024-04-12T22:02:55.000 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:54 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@grafana.a.service: Scheduled restart job, restart counter is at 7. 2024-04-12T22:02:55.000 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:54 smithi114 systemd[1]: Stopped Ceph grafana.a for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:02:55.001 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:54 smithi114 systemd[1]: Starting Ceph grafana.a for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:02:55.001 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:54 smithi114 podman[58103]: 2024-04-12 22:02:54.887197799 +0000 UTC m=+0.017502166 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-04-12T22:02:55.081 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:54 smithi097 podman[67254]: 2024-04-12 22:02:54.818554903 +0000 UTC m=+0.650738794 container remove 679b33a29a0cc1a76fb83b8403c75c795f938108068da55843d41c2882859362 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mon-c, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, ceph=True, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, name=centos-stream, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, release=754, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, GIT_CLEAN=True, io.openshift.expose-services=) 2024-04-12T22:02:55.082 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@mon.c.service: Deactivated successfully. 2024-04-12T22:02:55.082 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 systemd[1]: Stopped Ceph mon.c for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:02:55.082 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@mon.c.service: Consumed 7.762s CPU time. 2024-04-12T22:02:55.303 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:55 smithi114 podman[58103]: 2024-04-12 22:02:55.000216679 +0000 UTC m=+0.130521115 container create ce6684a8a4ab660fc9ab9e92f84622075794c7fa0ba1978878f71235685459da (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, vendor=Red Hat, Inc., vcs-type=git, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=236.1648460182, maintainer=Paul Cuzner , io.openshift.tags=base rhel8, distribution-scope=public, name=ubi8, com.redhat.component=ubi8-container, summary=Grafana Container configured for Ceph mgr/dashboard integration, version=8.5, io.openshift.expose-services=, io.k8s.display-name=Red Hat Universal Base Image 8, architecture=x86_64, description=Ceph Grafana Container, io.buildah.version=1.24.2, build-date=2022-03-28T10:36:18.413762, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI) 2024-04-12T22:02:55.410 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 systemd[1]: Starting Ceph mon.c for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:02:55.731 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 podman[67388]: 2024-04-12 22:02:55.381788096 +0000 UTC m=+0.017542777 image pull 668895944f0ec64b67a5b7564961d2756f2966e5d4b9338516300dbec9984951 quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a 2024-04-12T22:02:55.731 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 podman[67388]: 2024-04-12 22:02:55.503231188 +0000 UTC m=+0.138985861 container create 69507ba88603eec12ab2a5dadab336402e9cce7c6298eafd6aa1cd1b2988c12e (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mon-c, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-b277feb, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-12T22:02:55.743 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:55 smithi114 podman[58103]: 2024-04-12 22:02:55.302595706 +0000 UTC m=+0.432900081 container init ce6684a8a4ab660fc9ab9e92f84622075794c7fa0ba1978878f71235685459da (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Paul Cuzner , io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, architecture=x86_64, vendor=Red Hat, Inc., description=Ceph Grafana Container, release=236.1648460182, version=8.5, build-date=2022-03-28T10:36:18.413762, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.buildah.version=1.24.2, distribution-scope=public, com.redhat.component=ubi8-container, vcs-type=git, io.k8s.display-name=Red Hat Universal Base Image 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=ubi8) 2024-04-12T22:02:55.743 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:55 smithi114 podman[58103]: 2024-04-12 22:02:55.305280315 +0000 UTC m=+0.435584685 container start ce6684a8a4ab660fc9ab9e92f84622075794c7fa0ba1978878f71235685459da (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, distribution-scope=public, description=Ceph Grafana Container, io.openshift.expose-services=, io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , name=ubi8, io.openshift.tags=base rhel8, com.redhat.component=ubi8-container, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.buildah.version=1.24.2, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, release=236.1648460182, build-date=2022-03-28T10:36:18.413762, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., version=8.5, vcs-type=git, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182) 2024-04-12T22:02:55.743 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:55 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[58119]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-12T22:02:55.744 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:55 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[58119]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-12T22:02:55.744 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:55 smithi114 bash[58103]: ce6684a8a4ab660fc9ab9e92f84622075794c7fa0ba1978878f71235685459da 2024-04-12T22:02:55.744 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:55 smithi114 systemd[1]: Started Ceph grafana.a for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:02:55.744 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:55 smithi114 podman[58139]: 2024-04-12 22:02:55.393937487 +0000 UTC m=+0.052570865 container died ce6684a8a4ab660fc9ab9e92f84622075794c7fa0ba1978878f71235685459da (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-type=git, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.buildah.version=1.24.2, io.openshift.expose-services=, io.openshift.tags=base rhel8, version=8.5, description=Ceph Grafana Container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, release=236.1648460182, architecture=x86_64, io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8, com.redhat.component=ubi8-container, build-date=2022-03-28T10:36:18.413762, distribution-scope=public) 2024-04-12T22:02:56.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 podman[67388]: 2024-04-12 22:02:55.730787346 +0000 UTC m=+0.366542019 container init 69507ba88603eec12ab2a5dadab336402e9cce7c6298eafd6aa1cd1b2988c12e (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mon-c, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-b277feb, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-12T22:02:56.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 podman[67388]: 2024-04-12 22:02:55.733929143 +0000 UTC m=+0.369683825 container start 69507ba88603eec12ab2a5dadab336402e9cce7c6298eafd6aa1cd1b2988c12e (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mon-c, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-b277feb, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-12T22:02:56.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: set uid:gid to 167:167 (ceph:ceph) 2024-04-12T22:02:56.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev), process ceph-mon, pid 2 2024-04-12T22:02:56.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: pidfile_write: ignore empty --pid-file 2024-04-12T22:02:56.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: load: jerasure load: lrc 2024-04-12T22:02:56.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: RocksDB version: 7.9.2 2024-04-12T22:02:56.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Git sha 0 2024-04-12T22:02:56.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Compile date 2024-04-09 21:18:54 2024-04-12T22:02:56.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: DB SUMMARY 2024-04-12T22:02:56.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: DB Session ID: E96486P81WMRAZYW1T9Y 2024-04-12T22:02:56.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: CURRENT file: CURRENT 2024-04-12T22:02:56.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: IDENTITY file: IDENTITY 2024-04-12T22:02:56.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: MANIFEST file: MANIFEST-000009 size: 1092 Bytes 2024-04-12T22:02:56.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 1, files: 000027.sst 2024-04-12T22:02:56.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000025.log size: 324436 ; 2024-04-12T22:02:56.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.error_if_exists: 0 2024-04-12T22:02:56.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.create_if_missing: 0 2024-04-12T22:02:56.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.paranoid_checks: 1 2024-04-12T22:02:56.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-12T22:02:56.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-12T22:02:56.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-12T22:02:56.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.env: 0x55dd85f0fc60 2024-04-12T22:02:56.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.fs: PosixFileSystem 2024-04-12T22:02:56.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.info_log: 0x55dd87c71ae0 2024-04-12T22:02:56.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.max_file_opening_threads: 16 2024-04-12T22:02:56.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.statistics: (nil) 2024-04-12T22:02:56.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.use_fsync: 0 2024-04-12T22:02:56.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.max_log_file_size: 0 2024-04-12T22:02:56.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-12T22:02:56.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-12T22:02:56.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.keep_log_file_num: 1000 2024-04-12T22:02:56.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.recycle_log_file_num: 0 2024-04-12T22:02:56.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.allow_fallocate: 1 2024-04-12T22:02:56.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.allow_mmap_reads: 0 2024-04-12T22:02:56.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.allow_mmap_writes: 0 2024-04-12T22:02:56.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.use_direct_reads: 0 2024-04-12T22:02:56.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-12T22:02:56.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.create_missing_column_families: 0 2024-04-12T22:02:56.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.db_log_dir: 2024-04-12T22:02:56.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.wal_dir: 2024-04-12T22:02:56.164 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-12T22:02:56.164 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-12T22:02:56.164 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-12T22:02:56.164 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-12T22:02:56.164 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-12T22:02:56.164 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-12T22:02:56.164 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.advise_random_on_open: 1 2024-04-12T22:02:56.164 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.db_write_buffer_size: 0 2024-04-12T22:02:56.164 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.write_buffer_manager: 0x55dd87c44820 2024-04-12T22:02:56.164 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-12T22:02:56.164 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-12T22:02:56.164 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-12T22:02:56.164 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.rate_limiter: (nil) 2024-04-12T22:02:56.164 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-12T22:02:56.164 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.wal_recovery_mode: 2 2024-04-12T22:02:56.164 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.enable_thread_tracking: 0 2024-04-12T22:02:56.165 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.enable_pipelined_write: 0 2024-04-12T22:02:56.165 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.unordered_write: 0 2024-04-12T22:02:56.165 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-12T22:02:56.165 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-12T22:02:56.165 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-12T22:02:56.165 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-12T22:02:56.165 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.row_cache: None 2024-04-12T22:02:56.165 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.wal_filter: None 2024-04-12T22:02:56.165 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-12T22:02:56.166 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.allow_ingest_behind: 0 2024-04-12T22:02:56.166 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.two_write_queues: 0 2024-04-12T22:02:56.166 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.manual_wal_flush: 0 2024-04-12T22:02:56.166 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.wal_compression: 0 2024-04-12T22:02:56.166 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.atomic_flush: 0 2024-04-12T22:02:56.166 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-12T22:02:56.166 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-12T22:02:56.167 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-12T22:02:56.167 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.log_readahead_size: 0 2024-04-12T22:02:56.167 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-12T22:02:56.167 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.best_efforts_recovery: 0 2024-04-12T22:02:56.167 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-12T22:02:56.167 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-12T22:02:56.167 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.allow_data_in_errors: 0 2024-04-12T22:02:56.167 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.db_host_id: __hostname__ 2024-04-12T22:02:56.167 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-12T22:02:56.167 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.max_background_jobs: 2 2024-04-12T22:02:56.167 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.max_background_compactions: -1 2024-04-12T22:02:56.167 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.max_subcompactions: 1 2024-04-12T22:02:56.167 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-12T22:02:56.167 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-12T22:02:56.167 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-12T22:02:56.167 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.max_total_wal_size: 0 2024-04-12T22:02:56.168 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-12T22:02:56.168 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-12T22:02:56.168 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-12T22:02:56.168 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-12T22:02:56.168 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.max_open_files: -1 2024-04-12T22:02:56.168 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.bytes_per_sync: 0 2024-04-12T22:02:56.168 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-12T22:02:56.168 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-12T22:02:56.168 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.compaction_readahead_size: 0 2024-04-12T22:02:56.168 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.max_background_flushes: -1 2024-04-12T22:02:56.168 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Compression algorithms supported: 2024-04-12T22:02:56.168 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-12T22:02:56.168 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: kZSTD supported: 0 2024-04-12T22:02:56.168 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: kXpressCompression supported: 0 2024-04-12T22:02:56.168 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: kLZ4HCCompression supported: 1 2024-04-12T22:02:56.168 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: kZlibCompression supported: 1 2024-04-12T22:02:56.169 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: kSnappyCompression supported: 1 2024-04-12T22:02:56.169 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: kLZ4Compression supported: 1 2024-04-12T22:02:56.169 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: kBZip2Compression supported: 0 2024-04-12T22:02:56.169 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-12T22:02:56.169 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-12T22:02:56.169 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 2024-04-12T22:02:56.169 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-12T22:02:56.169 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-12T22:02:56.169 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.merge_operator: 2024-04-12T22:02:56.169 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.compaction_filter: None 2024-04-12T22:02:56.169 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.compaction_filter_factory: None 2024-04-12T22:02:56.169 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.sst_partitioner_factory: None 2024-04-12T22:02:56.169 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-12T22:02:56.169 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-12T22:02:56.169 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55dd87c71be0) 2024-04-12T22:02:56.169 INFO:journalctl@ceph.mon.c.smithi097.stdout: cache_index_and_filter_blocks: 1 2024-04-12T22:02:56.170 INFO:journalctl@ceph.mon.c.smithi097.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-12T22:02:56.170 INFO:journalctl@ceph.mon.c.smithi097.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-12T22:02:56.170 INFO:journalctl@ceph.mon.c.smithi097.stdout: pin_top_level_index_and_filter: 1 2024-04-12T22:02:56.170 INFO:journalctl@ceph.mon.c.smithi097.stdout: index_type: 0 2024-04-12T22:02:56.170 INFO:journalctl@ceph.mon.c.smithi097.stdout: data_block_index_type: 0 2024-04-12T22:02:56.170 INFO:journalctl@ceph.mon.c.smithi097.stdout: index_shortening: 1 2024-04-12T22:02:56.170 INFO:journalctl@ceph.mon.c.smithi097.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-12T22:02:56.170 INFO:journalctl@ceph.mon.c.smithi097.stdout: checksum: 4 2024-04-12T22:02:56.170 INFO:journalctl@ceph.mon.c.smithi097.stdout: no_block_cache: 0 2024-04-12T22:02:56.170 INFO:journalctl@ceph.mon.c.smithi097.stdout: block_cache: 0x55dd87c75090 2024-04-12T22:02:56.170 INFO:journalctl@ceph.mon.c.smithi097.stdout: block_cache_name: BinnedLRUCache 2024-04-12T22:02:56.170 INFO:journalctl@ceph.mon.c.smithi097.stdout: block_cache_options: 2024-04-12T22:02:56.170 INFO:journalctl@ceph.mon.c.smithi097.stdout: capacity : 536870912 2024-04-12T22:02:56.170 INFO:journalctl@ceph.mon.c.smithi097.stdout: num_shard_bits : 4 2024-04-12T22:02:56.170 INFO:journalctl@ceph.mon.c.smithi097.stdout: strict_capacity_limit : 0 2024-04-12T22:02:56.171 INFO:journalctl@ceph.mon.c.smithi097.stdout: high_pri_pool_ratio: 0.000 2024-04-12T22:02:56.171 INFO:journalctl@ceph.mon.c.smithi097.stdout: block_cache_compressed: (nil) 2024-04-12T22:02:56.171 INFO:journalctl@ceph.mon.c.smithi097.stdout: persistent_cache: (nil) 2024-04-12T22:02:56.171 INFO:journalctl@ceph.mon.c.smithi097.stdout: block_size: 4096 2024-04-12T22:02:56.171 INFO:journalctl@ceph.mon.c.smithi097.stdout: block_size_deviation: 10 2024-04-12T22:02:56.171 INFO:journalctl@ceph.mon.c.smithi097.stdout: block_restart_interval: 16 2024-04-12T22:02:56.171 INFO:journalctl@ceph.mon.c.smithi097.stdout: index_block_restart_interval: 1 2024-04-12T22:02:56.171 INFO:journalctl@ceph.mon.c.smithi097.stdout: metadata_block_size: 4096 2024-04-12T22:02:56.171 INFO:journalctl@ceph.mon.c.smithi097.stdout: partition_filters: 0 2024-04-12T22:02:56.171 INFO:journalctl@ceph.mon.c.smithi097.stdout: use_delta_encoding: 1 2024-04-12T22:02:56.171 INFO:journalctl@ceph.mon.c.smithi097.stdout: filter_policy: bloomfilter 2024-04-12T22:02:56.171 INFO:journalctl@ceph.mon.c.smithi097.stdout: whole_key_filtering: 1 2024-04-12T22:02:56.171 INFO:journalctl@ceph.mon.c.smithi097.stdout: verify_compression: 0 2024-04-12T22:02:56.171 INFO:journalctl@ceph.mon.c.smithi097.stdout: read_amp_bytes_per_bit: 0 2024-04-12T22:02:56.171 INFO:journalctl@ceph.mon.c.smithi097.stdout: format_version: 5 2024-04-12T22:02:56.171 INFO:journalctl@ceph.mon.c.smithi097.stdout: enable_index_compression: 1 2024-04-12T22:02:56.172 INFO:journalctl@ceph.mon.c.smithi097.stdout: block_align: 0 2024-04-12T22:02:56.172 INFO:journalctl@ceph.mon.c.smithi097.stdout: max_auto_readahead_size: 262144 2024-04-12T22:02:56.172 INFO:journalctl@ceph.mon.c.smithi097.stdout: prepopulate_block_cache: 0 2024-04-12T22:02:56.172 INFO:journalctl@ceph.mon.c.smithi097.stdout: initial_auto_readahead_size: 8192 2024-04-12T22:02:56.172 INFO:journalctl@ceph.mon.c.smithi097.stdout: num_file_reads_for_auto_readahead: 2 2024-04-12T22:02:56.172 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.write_buffer_size: 33554432 2024-04-12T22:02:56.172 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.max_write_buffer_number: 2 2024-04-12T22:02:56.172 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.compression: NoCompression 2024-04-12T22:02:56.172 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.bottommost_compression: Disabled 2024-04-12T22:02:56.172 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.prefix_extractor: nullptr 2024-04-12T22:02:56.172 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-12T22:02:56.172 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.num_levels: 7 2024-04-12T22:02:56.172 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-12T22:02:56.172 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-12T22:02:56.172 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-12T22:02:56.172 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-12T22:02:56.173 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-12T22:02:56.173 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-12T22:02:56.173 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-12T22:02:56.173 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-12T22:02:56.173 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-12T22:02:56.173 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-12T22:02:56.173 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-12T22:02:56.173 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-12T22:02:56.173 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-12T22:02:56.173 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.compression_opts.level: 32767 2024-04-12T22:02:56.173 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.compression_opts.strategy: 0 2024-04-12T22:02:56.173 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-12T22:02:56.173 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-12T22:02:56.173 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-12T22:02:56.174 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-12T22:02:56.174 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.compression_opts.enabled: false 2024-04-12T22:02:56.174 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-12T22:02:56.174 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-12T22:02:56.174 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-12T22:02:56.174 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-12T22:02:56.174 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.target_file_size_base: 67108864 2024-04-12T22:02:56.174 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-12T22:02:56.174 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-12T22:02:56.174 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-12T22:02:56.174 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-12T22:02:56.174 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-12T22:02:56.175 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-12T22:02:56.175 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-12T22:02:56.175 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-12T22:02:56.175 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-12T22:02:56.175 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-12T22:02:56.175 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-12T22:02:56.175 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-12T22:02:56.175 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-12T22:02:56.175 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-12T22:02:56.175 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.arena_block_size: 1048576 2024-04-12T22:02:56.175 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-12T22:02:56.175 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-12T22:02:56.176 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.disable_auto_compactions: 0 2024-04-12T22:02:56.176 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-12T22:02:56.176 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-12T22:02:56.176 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-12T22:02:56.176 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-12T22:02:56.176 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-12T22:02:56.176 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-12T22:02:56.176 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-12T22:02:56.176 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-12T22:02:56.176 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-12T22:02:56.176 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-12T22:02:56.176 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.table_properties_collectors: 2024-04-12T22:02:56.176 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.inplace_update_support: 0 2024-04-12T22:02:56.176 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-12T22:02:56.176 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-12T22:02:56.177 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-12T22:02:56.177 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-12T22:02:56.177 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.bloom_locality: 0 2024-04-12T22:02:56.177 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.max_successive_merges: 0 2024-04-12T22:02:56.177 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-12T22:02:56.177 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.paranoid_file_checks: 0 2024-04-12T22:02:56.177 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.force_consistency_checks: 1 2024-04-12T22:02:56.177 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.report_bg_io_stats: 0 2024-04-12T22:02:56.177 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.ttl: 2592000 2024-04-12T22:02:56.177 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-12T22:02:56.177 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-12T22:02:56.177 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-12T22:02:56.177 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.enable_blob_files: false 2024-04-12T22:02:56.177 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.min_blob_size: 0 2024-04-12T22:02:56.177 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.blob_file_size: 268435456 2024-04-12T22:02:56.177 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-12T22:02:56.178 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-12T22:02:56.178 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-12T22:02:56.178 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-12T22:02:56.178 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-12T22:02:56.178 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.blob_file_starting_level: 0 2024-04-12T22:02:56.178 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-12T22:02:56.178 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 27.sst 2024-04-12T22:02:56.178 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-12T22:02:56.178 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 29, last_sequence is 15173, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-12T22:02:56.178 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-04-12T22:02:56.178 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 843dc4ac-3e8d-43fe-833b-e7956a78dc14 2024-04-12T22:02:56.178 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712959375763958, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-04-12T22:02:56.178 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-04-12T22:02:56.178 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712959375765335, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 216073, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 15178, "largest_seqno": 15310, "table_properties": {"data_size": 214380, "index_size": 489, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 197, "raw_key_size": 2025, "raw_average_key_size": 26, "raw_value_size": 212563, "raw_average_value_size": 2796, "num_data_blocks": 20, "num_entries": 76, "num_filter_entries": 76, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1712959375, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "843dc4ac-3e8d-43fe-833b-e7956a78dc14", "db_session_id": "E96486P81WMRAZYW1T9Y", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-04-12T22:02:56.178 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712959375765416, "job": 1, "event": "recovery_finished"} 2024-04-12T22:02:56.178 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-04-12T22:02:56.179 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-12T22:02:56.179 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-12T22:02:56.179 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55dd87d66000 2024-04-12T22:02:56.179 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: DB pointer 0x55dd87d50000 2024-04-12T22:02:56.179 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-12T22:02:56.179 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-12T22:02:56.179 INFO:journalctl@ceph.mon.c.smithi097.stdout: ** DB Stats ** 2024-04-12T22:02:56.179 INFO:journalctl@ceph.mon.c.smithi097.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-12T22:02:56.179 INFO:journalctl@ceph.mon.c.smithi097.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-12T22:02:56.179 INFO:journalctl@ceph.mon.c.smithi097.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-12T22:02:56.179 INFO:journalctl@ceph.mon.c.smithi097.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-12T22:02:56.179 INFO:journalctl@ceph.mon.c.smithi097.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-12T22:02:56.179 INFO:journalctl@ceph.mon.c.smithi097.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-12T22:02:56.179 INFO:journalctl@ceph.mon.c.smithi097.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-12T22:02:56.179 INFO:journalctl@ceph.mon.c.smithi097.stdout: 2024-04-12T22:02:56.179 INFO:journalctl@ceph.mon.c.smithi097.stdout: ** Compaction Stats [default] ** 2024-04-12T22:02:56.180 INFO:journalctl@ceph.mon.c.smithi097.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-12T22:02:56.180 INFO:journalctl@ceph.mon.c.smithi097.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-12T22:02:56.180 INFO:journalctl@ceph.mon.c.smithi097.stdout: L0 1/0 211.01 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 260.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-12T22:02:56.180 INFO:journalctl@ceph.mon.c.smithi097.stdout: L6 1/0 11.56 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-12T22:02:56.180 INFO:journalctl@ceph.mon.c.smithi097.stdout: Sum 2/0 11.76 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 260.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-12T22:02:56.180 INFO:journalctl@ceph.mon.c.smithi097.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 260.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-12T22:02:56.180 INFO:journalctl@ceph.mon.c.smithi097.stdout: 2024-04-12T22:02:56.180 INFO:journalctl@ceph.mon.c.smithi097.stdout: ** Compaction Stats [default] ** 2024-04-12T22:02:56.180 INFO:journalctl@ceph.mon.c.smithi097.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-12T22:02:56.180 INFO:journalctl@ceph.mon.c.smithi097.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-12T22:02:56.180 INFO:journalctl@ceph.mon.c.smithi097.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 260.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-12T22:02:56.180 INFO:journalctl@ceph.mon.c.smithi097.stdout: 2024-04-12T22:02:56.180 INFO:journalctl@ceph.mon.c.smithi097.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-12T22:02:56.180 INFO:journalctl@ceph.mon.c.smithi097.stdout: 2024-04-12T22:02:56.180 INFO:journalctl@ceph.mon.c.smithi097.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-12T22:02:56.180 INFO:journalctl@ceph.mon.c.smithi097.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-12T22:02:56.181 INFO:journalctl@ceph.mon.c.smithi097.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-12T22:02:56.181 INFO:journalctl@ceph.mon.c.smithi097.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-12T22:02:56.181 INFO:journalctl@ceph.mon.c.smithi097.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-12T22:02:56.181 INFO:journalctl@ceph.mon.c.smithi097.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-12T22:02:56.181 INFO:journalctl@ceph.mon.c.smithi097.stdout: Cumulative compaction: 0.00 GB write, 61.49 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-12T22:02:56.181 INFO:journalctl@ceph.mon.c.smithi097.stdout: Interval compaction: 0.00 GB write, 61.49 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-12T22:02:56.181 INFO:journalctl@ceph.mon.c.smithi097.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-12T22:02:56.181 INFO:journalctl@ceph.mon.c.smithi097.stdout: Block cache BinnedLRUCache@0x55dd87c75090#2 capacity: 512.00 MB usage: 0.81 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-04-12T22:02:56.181 INFO:journalctl@ceph.mon.c.smithi097.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.23 KB,4.47035e-05%) IndexBlock(1,0.58 KB,0.000110269%) Misc(1,0.00 KB,0%) 2024-04-12T22:02:56.181 INFO:journalctl@ceph.mon.c.smithi097.stdout: 2024-04-12T22:02:56.181 INFO:journalctl@ceph.mon.c.smithi097.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-12T22:02:56.181 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: starting mon.c rank 1 at public addrs [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] at bind addrs [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 80bf7680-f916-11ee-bc8d-c7b262605968 2024-04-12T22:02:56.181 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: mon.c@-1(???) e3 preinit fsid 80bf7680-f916-11ee-bc8d-c7b262605968 2024-04-12T22:02:56.181 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: mon.c@-1(???).mds e1 new map 2024-04-12T22:02:56.181 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: mon.c@-1(???).mds e1 print_map 2024-04-12T22:02:56.182 INFO:journalctl@ceph.mon.c.smithi097.stdout: e1 2024-04-12T22:02:56.182 INFO:journalctl@ceph.mon.c.smithi097.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T22:02:56.182 INFO:journalctl@ceph.mon.c.smithi097.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T22:02:56.182 INFO:journalctl@ceph.mon.c.smithi097.stdout: legacy client fscid: -1 2024-04-12T22:02:56.182 INFO:journalctl@ceph.mon.c.smithi097.stdout: 2024-04-12T22:02:56.182 INFO:journalctl@ceph.mon.c.smithi097.stdout: No filesystems configured 2024-04-12T22:02:56.182 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: mon.c@-1(???).osd e106 crush map has features 3314933000854323200, adjusting msgr requires 2024-04-12T22:02:56.182 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: mon.c@-1(???).osd e106 crush map has features 432629239337189376, adjusting msgr requires 2024-04-12T22:02:56.182 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: mon.c@-1(???).osd e106 crush map has features 432629239337189376, adjusting msgr requires 2024-04-12T22:02:56.182 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: mon.c@-1(???).osd e106 crush map has features 432629239337189376, adjusting msgr requires 2024-04-12T22:02:56.182 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 ceph-mon[67407]: mon.c@-1(???).paxosservice(auth 1..23) refresh upgraded, format 0 -> 3 2024-04-12T22:02:56.182 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 bash[67388]: 69507ba88603eec12ab2a5dadab336402e9cce7c6298eafd6aa1cd1b2988c12e 2024-04-12T22:02:56.182 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:55 smithi097 systemd[1]: Started Ceph mon.c for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:02:56.243 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:55 smithi114 podman[58139]: 2024-04-12 22:02:55.96485116 +0000 UTC m=+0.623484529 container remove ce6684a8a4ab660fc9ab9e92f84622075794c7fa0ba1978878f71235685459da (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, io.openshift.tags=base rhel8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.24.2, name=ubi8, com.redhat.component=ubi8-container, distribution-scope=public, version=8.5, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, architecture=x86_64, io.openshift.expose-services=, build-date=2022-03-28T10:36:18.413762, description=Ceph Grafana Container, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, vcs-type=git, maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, release=236.1648460182) 2024-04-12T22:02:56.243 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:55 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-04-12T22:02:56.243 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:02:55 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-04-12T22:02:58.485 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:58 smithi097 ceph-mon[64853]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-12T22:02:58.485 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:58 smithi097 ceph-mon[64853]: mon.c calling monitor election 2024-04-12T22:02:58.485 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:58 smithi097 ceph-mon[64853]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-12T22:02:58.485 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:58 smithi097 ceph-mon[64853]: mon.a calling monitor election 2024-04-12T22:02:58.485 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:58 smithi097 ceph-mon[64853]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-12T22:02:58.485 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:58 smithi097 ceph-mon[64853]: monmap epoch 3 2024-04-12T22:02:58.485 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:58 smithi097 ceph-mon[64853]: fsid 80bf7680-f916-11ee-bc8d-c7b262605968 2024-04-12T22:02:58.485 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:58 smithi097 ceph-mon[64853]: last_changed 2024-04-12T21:51:25.246349+0000 2024-04-12T22:02:58.486 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:58 smithi097 ceph-mon[64853]: created 2024-04-12T21:49:45.534140+0000 2024-04-12T22:02:58.486 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:58 smithi097 ceph-mon[64853]: min_mon_release 17 (quincy) 2024-04-12T22:02:58.486 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:58 smithi097 ceph-mon[64853]: election_strategy: 1 2024-04-12T22:02:58.486 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:58 smithi097 ceph-mon[64853]: 0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2024-04-12T22:02:58.486 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:58 smithi097 ceph-mon[64853]: 1: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2024-04-12T22:02:58.486 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:58 smithi097 ceph-mon[64853]: 2: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-04-12T22:02:58.486 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:58 smithi097 ceph-mon[64853]: fsmap 2024-04-12T22:02:58.486 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:58 smithi097 ceph-mon[64853]: osdmap e106: 8 total, 8 up, 8 in 2024-04-12T22:02:58.486 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:58 smithi097 ceph-mon[64853]: mgrmap e38: y(active, since 17s), standbys: x 2024-04-12T22:02:58.486 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:58 smithi097 ceph-mon[64853]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-12T22:02:58.486 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:58 smithi097 ceph-mon[64853]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-12T22:02:58.486 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:58 smithi097 ceph-mon[64853]: daemon grafana.a on smithi114 is in unknown state 2024-04-12T22:02:58.486 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:58 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:02:58.487 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:58 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:02:58.487 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:58 smithi097 ceph-mon[67407]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-12T22:02:58.487 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:58 smithi097 ceph-mon[67407]: mon.c calling monitor election 2024-04-12T22:02:58.487 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:58 smithi097 ceph-mon[67407]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-12T22:02:58.487 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:58 smithi097 ceph-mon[67407]: mon.a calling monitor election 2024-04-12T22:02:58.487 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:58 smithi097 ceph-mon[67407]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-12T22:02:58.487 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:58 smithi097 ceph-mon[67407]: monmap epoch 3 2024-04-12T22:02:58.487 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:58 smithi097 ceph-mon[67407]: fsid 80bf7680-f916-11ee-bc8d-c7b262605968 2024-04-12T22:02:58.487 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:58 smithi097 ceph-mon[67407]: last_changed 2024-04-12T21:51:25.246349+0000 2024-04-12T22:02:58.487 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:58 smithi097 ceph-mon[67407]: created 2024-04-12T21:49:45.534140+0000 2024-04-12T22:02:58.487 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:58 smithi097 ceph-mon[67407]: min_mon_release 17 (quincy) 2024-04-12T22:02:58.487 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:58 smithi097 ceph-mon[67407]: election_strategy: 1 2024-04-12T22:02:58.488 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:58 smithi097 ceph-mon[67407]: 0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2024-04-12T22:02:58.488 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:58 smithi097 ceph-mon[67407]: 1: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2024-04-12T22:02:58.488 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:58 smithi097 ceph-mon[67407]: 2: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-04-12T22:02:58.488 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:58 smithi097 ceph-mon[67407]: fsmap 2024-04-12T22:02:58.488 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:58 smithi097 ceph-mon[67407]: osdmap e106: 8 total, 8 up, 8 in 2024-04-12T22:02:58.488 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:58 smithi097 ceph-mon[67407]: mgrmap e38: y(active, since 17s), standbys: x 2024-04-12T22:02:58.488 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:58 smithi097 ceph-mon[67407]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-12T22:02:58.488 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:58 smithi097 ceph-mon[67407]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-12T22:02:58.488 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:58 smithi097 ceph-mon[67407]: daemon grafana.a on smithi114 is in unknown state 2024-04-12T22:02:58.488 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:58 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:02:58.488 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:58 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:02:58.661 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:58 smithi114 ceph-mon[25759]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-12T22:02:58.661 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:58 smithi114 ceph-mon[25759]: mon.c calling monitor election 2024-04-12T22:02:58.661 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:58 smithi114 ceph-mon[25759]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-12T22:02:58.661 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:58 smithi114 ceph-mon[25759]: mon.a calling monitor election 2024-04-12T22:02:58.661 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:58 smithi114 ceph-mon[25759]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-12T22:02:58.661 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:58 smithi114 ceph-mon[25759]: monmap epoch 3 2024-04-12T22:02:58.661 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:58 smithi114 ceph-mon[25759]: fsid 80bf7680-f916-11ee-bc8d-c7b262605968 2024-04-12T22:02:58.661 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:58 smithi114 ceph-mon[25759]: last_changed 2024-04-12T21:51:25.246349+0000 2024-04-12T22:02:58.662 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:58 smithi114 ceph-mon[25759]: created 2024-04-12T21:49:45.534140+0000 2024-04-12T22:02:58.662 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:58 smithi114 ceph-mon[25759]: min_mon_release 17 (quincy) 2024-04-12T22:02:58.662 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:58 smithi114 ceph-mon[25759]: election_strategy: 1 2024-04-12T22:02:58.662 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:58 smithi114 ceph-mon[25759]: 0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2024-04-12T22:02:58.662 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:58 smithi114 ceph-mon[25759]: 1: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2024-04-12T22:02:58.662 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:58 smithi114 ceph-mon[25759]: 2: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-04-12T22:02:58.662 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:58 smithi114 ceph-mon[25759]: fsmap 2024-04-12T22:02:58.662 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:58 smithi114 ceph-mon[25759]: osdmap e106: 8 total, 8 up, 8 in 2024-04-12T22:02:58.662 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:58 smithi114 ceph-mon[25759]: mgrmap e38: y(active, since 17s), standbys: x 2024-04-12T22:02:58.662 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:58 smithi114 ceph-mon[25759]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-12T22:02:58.662 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:58 smithi114 ceph-mon[25759]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-12T22:02:58.662 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:58 smithi114 ceph-mon[25759]: daemon grafana.a on smithi114 is in unknown state 2024-04-12T22:02:58.662 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:58 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:02:58.662 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:58 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:02:58.993 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:02:58 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:02:58.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"smithi097:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"172.21.15.114:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-12T22:02:59.381 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:02:59 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:02:59] "GET /metrics HTTP/1.1" 200 38253 "" "Prometheus/2.33.4" 2024-04-12T22:02:59.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:02:59 smithi097 ceph-mon[64853]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 964 B/s rd, 0 op/s 2024-04-12T22:02:59.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:02:59 smithi097 ceph-mon[67407]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 964 B/s rd, 0 op/s 2024-04-12T22:02:59.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:02:59 smithi114 ceph-mon[25759]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 964 B/s rd, 0 op/s 2024-04-12T22:03:00.910 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:03:00 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:03:00.773+0000 7fe40fa5a700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-12T22:03:01.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:01 smithi097 ceph-mon[64853]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 964 B/s rd, 0 op/s 2024-04-12T22:03:01.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:01 smithi097 ceph-mon[64853]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:03:01.910 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:01 smithi097 ceph-mon[67407]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 964 B/s rd, 0 op/s 2024-04-12T22:03:01.910 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:01 smithi097 ceph-mon[67407]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:03:02.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:01 smithi114 ceph-mon[25759]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 964 B/s rd, 0 op/s 2024-04-12T22:03:02.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:01 smithi114 ceph-mon[25759]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:03:03.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:02 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:03.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:02 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:03.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:02 smithi114 ceph-mon[25759]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:03:03.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:02 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:03.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:02 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:03.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:02 smithi097 ceph-mon[64853]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:03:03.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:02 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:03.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:02 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:03.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:02 smithi097 ceph-mon[67407]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:03:04.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:03 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:04.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:03 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:04.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:03 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:04.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:03 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:04.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:03 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:04.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:03 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:05.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:04 smithi097 ceph-mon[64853]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:03:05.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:04 smithi097 ceph-mon[67407]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:03:05.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:04 smithi114 ceph-mon[25759]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:03:06.334 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:05 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@grafana.a.service: Scheduled restart job, restart counter is at 8. 2024-04-12T22:03:06.334 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:05 smithi114 systemd[1]: Stopped Ceph grafana.a for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:03:06.334 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:06 smithi114 systemd[1]: Starting Ceph grafana.a for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:03:06.334 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:06 smithi114 podman[58217]: 2024-04-12 22:03:06.131585409 +0000 UTC m=+0.017715497 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-04-12T22:03:06.334 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:06 smithi114 podman[58217]: 2024-04-12 22:03:06.232472179 +0000 UTC m=+0.118602237 container create dd3406f3cee88b9cec3f796425034f43603bd29b7a3e81a9103a5cb35bafe14a (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, io.openshift.tags=base rhel8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, description=Ceph Grafana Container, name=ubi8, vendor=Red Hat, Inc., vcs-type=git, maintainer=Paul Cuzner , io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.component=ubi8-container, version=8.5, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.buildah.version=1.24.2, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, build-date=2022-03-28T10:36:18.413762, release=236.1648460182, distribution-scope=public) 2024-04-12T22:03:06.632 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:06 smithi114 podman[58217]: 2024-04-12 22:03:06.493122241 +0000 UTC m=+0.379252300 container init dd3406f3cee88b9cec3f796425034f43603bd29b7a3e81a9103a5cb35bafe14a (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.24.2, com.redhat.component=ubi8-container, name=ubi8, version=8.5, build-date=2022-03-28T10:36:18.413762, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Grafana Container configured for Ceph mgr/dashboard integration, release=236.1648460182, vcs-type=git, maintainer=Paul Cuzner , vendor=Red Hat, Inc., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.display-name=Red Hat Universal Base Image 8, architecture=x86_64, io.openshift.tags=base rhel8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container, io.openshift.expose-services=) 2024-04-12T22:03:06.632 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:06 smithi114 podman[58217]: 2024-04-12 22:03:06.495984329 +0000 UTC m=+0.382114398 container start dd3406f3cee88b9cec3f796425034f43603bd29b7a3e81a9103a5cb35bafe14a (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, build-date=2022-03-28T10:36:18.413762, release=236.1648460182, distribution-scope=public, name=ubi8, maintainer=Paul Cuzner , architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container, io.openshift.tags=base rhel8, com.redhat.component=ubi8-container, version=8.5, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.24.2, io.openshift.expose-services=, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56) 2024-04-12T22:03:06.632 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:06 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[58231]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-12T22:03:06.632 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:06 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[58231]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-12T22:03:06.993 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:06 smithi114 bash[58217]: dd3406f3cee88b9cec3f796425034f43603bd29b7a3e81a9103a5cb35bafe14a 2024-04-12T22:03:06.993 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:06 smithi114 systemd[1]: Started Ceph grafana.a for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:03:06.993 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:06 smithi114 podman[58347]: 2024-04-12 22:03:06.788157473 +0000 UTC m=+0.252535117 container died dd3406f3cee88b9cec3f796425034f43603bd29b7a3e81a9103a5cb35bafe14a (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.24.2, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, maintainer=Paul Cuzner , name=ubi8, release=236.1648460182, com.redhat.component=ubi8-container, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, distribution-scope=public, description=Ceph Grafana Container, vendor=Red Hat, Inc., architecture=x86_64, build-date=2022-03-28T10:36:18.413762, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-04-12T22:03:07.365 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:07 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:07.365 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:07 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:07.365 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:07 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:07.365 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:07 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:03:07.365 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:07 smithi114 ceph-mon[25759]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:03:07.365 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:07 smithi114 ceph-mon[25759]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2024-04-12T22:03:07.365 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:07 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:07.365 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:07 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:03:07.365 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:07 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:03:07.365 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:07 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-12T22:03:07.365 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:07 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-04-12T22:03:07.365 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:07 smithi114 ceph-mon[25759]: Upgrade: It appears safe to stop mon.b 2024-04-12T22:03:07.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:07 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:07.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:07 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:07.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:07 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:07.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:07 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:03:07.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:07 smithi097 ceph-mon[67407]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:03:07.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:07 smithi097 ceph-mon[67407]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2024-04-12T22:03:07.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:07 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:07.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:07 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:03:07.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:07 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:03:07.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:07 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-12T22:03:07.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:07 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-04-12T22:03:07.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:07 smithi097 ceph-mon[67407]: Upgrade: It appears safe to stop mon.b 2024-04-12T22:03:07.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:07 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:07.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:07 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:07.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:07 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:07.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:07 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:03:07.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:07 smithi097 ceph-mon[64853]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:03:07.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:07 smithi097 ceph-mon[64853]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2024-04-12T22:03:07.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:07 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:07.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:07 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:03:07.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:07 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:03:07.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:07 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-12T22:03:07.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:07 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-04-12T22:03:07.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:07 smithi097 ceph-mon[64853]: Upgrade: It appears safe to stop mon.b 2024-04-12T22:03:07.675 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:07 smithi114 podman[58347]: 2024-04-12 22:03:07.364066935 +0000 UTC m=+0.828444581 container remove dd3406f3cee88b9cec3f796425034f43603bd29b7a3e81a9103a5cb35bafe14a (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, architecture=x86_64, maintainer=Paul Cuzner , com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.24.2, version=8.5, com.redhat.component=ubi8-container, description=Ceph Grafana Container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, release=236.1648460182, vcs-type=git, vendor=Red Hat, Inc., io.openshift.expose-services=, build-date=2022-03-28T10:36:18.413762, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=ubi8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.tags=base rhel8) 2024-04-12T22:03:07.675 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:07 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-04-12T22:03:07.675 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:07 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-04-12T22:03:08.743 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:03:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:03:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"smithi097:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"172.21.15.114:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-12T22:03:09.540 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:03:09 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:03:09] "GET /metrics HTTP/1.1" 200 38253 "" "Prometheus/2.33.4" 2024-04-12T22:03:09.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:09 smithi114 ceph-mon[25759]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2024-04-12T22:03:09.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:09 smithi114 ceph-mon[25759]: Upgrade: Updating mon.b 2024-04-12T22:03:09.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:09 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:09.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:09 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T22:03:09.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:09 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T22:03:09.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:09 smithi114 ceph-mon[25759]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:09.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:09 smithi114 ceph-mon[25759]: Deploying daemon mon.b on smithi114 2024-04-12T22:03:09.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:09 smithi097 ceph-mon[64853]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2024-04-12T22:03:09.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:09 smithi097 ceph-mon[64853]: Upgrade: Updating mon.b 2024-04-12T22:03:09.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:09 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:09.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:09 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T22:03:09.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:09 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T22:03:09.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:09 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:09.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:09 smithi097 ceph-mon[64853]: Deploying daemon mon.b on smithi114 2024-04-12T22:03:09.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:09 smithi097 ceph-mon[67407]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2024-04-12T22:03:09.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:09 smithi097 ceph-mon[67407]: Upgrade: Updating mon.b 2024-04-12T22:03:09.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:09 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:09.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:09 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T22:03:09.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:09 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T22:03:09.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:09 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:09.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:09 smithi097 ceph-mon[67407]: Deploying daemon mon.b on smithi114 2024-04-12T22:03:10.475 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:10 smithi114 systemd[1]: Stopping Ceph mon.b for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:03:10.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:10 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mon-b[25742]: 2024-04-12T22:03:10.473+0000 7f4df48f2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-12T22:03:10.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:10 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mon-b[25742]: 2024-04-12T22:03:10.473+0000 7f4df48f2700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-12T22:03:10.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:10 smithi114 podman[58617]: 2024-04-12 22:03:10.708459554 +0000 UTC m=+0.351034945 container died 47a08e70f856e75149a04c8cb8f8c88af766d0777608c1ed8b8a247d3b52cc5b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mon-b, io.openshift.tags=base centos centos-stream, distribution-scope=public, release=754, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, name=centos-stream, GIT_BRANCH=HEAD, RELEASE=HEAD, ceph=True, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container) 2024-04-12T22:03:11.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:11 smithi114 podman[58617]: 2024-04-12 22:03:11.048501887 +0000 UTC m=+0.691077278 container cleanup 47a08e70f856e75149a04c8cb8f8c88af766d0777608c1ed8b8a247d3b52cc5b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mon-b, release=754, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., distribution-scope=public, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, maintainer=Guillaume Abrioux , io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, name=centos-stream, io.buildah.version=1.19.8) 2024-04-12T22:03:11.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:11 smithi114 bash[58617]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mon-b 2024-04-12T22:03:11.666 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:11 smithi114 podman[58631]: 2024-04-12 22:03:11.249843382 +0000 UTC m=+0.542834001 container remove 47a08e70f856e75149a04c8cb8f8c88af766d0777608c1ed8b8a247d3b52cc5b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mon-b, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, distribution-scope=public, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, vcs-type=git, build-date=2022-05-03T08:36:31.336870, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, version=8, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream) 2024-04-12T22:03:11.666 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:11 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@mon.b.service: Deactivated successfully. 2024-04-12T22:03:11.666 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:11 smithi114 systemd[1]: Stopped Ceph mon.b for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:03:11.666 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:11 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@mon.b.service: Consumed 9.181s CPU time. 2024-04-12T22:03:11.957 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:11 smithi114 systemd[1]: Starting Ceph mon.b for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:03:11.957 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:11 smithi114 podman[58761]: 2024-04-12 22:03:11.86771785 +0000 UTC m=+0.103067384 container create 71a2eff159a33616171e9ceaec3ab3a21f9f32db7fe3fb91524b7097f74f0d55 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mon-b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b277feb, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-12T22:03:11.957 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:11 smithi114 podman[58761]: 2024-04-12 22:03:11.782996548 +0000 UTC m=+0.018346084 image pull 668895944f0ec64b67a5b7564961d2756f2966e5d4b9338516300dbec9984951 quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a 2024-04-12T22:03:12.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 podman[58761]: 2024-04-12 22:03:12.062631892 +0000 UTC m=+0.297981438 container init 71a2eff159a33616171e9ceaec3ab3a21f9f32db7fe3fb91524b7097f74f0d55 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mon-b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-b277feb, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD) 2024-04-12T22:03:12.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 podman[58761]: 2024-04-12 22:03:12.066540663 +0000 UTC m=+0.301890199 container start 71a2eff159a33616171e9ceaec3ab3a21f9f32db7fe3fb91524b7097f74f0d55 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, RELEASE=squid-b277feb, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-12T22:03:12.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: set uid:gid to 167:167 (ceph:ceph) 2024-04-12T22:03:12.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev), process ceph-mon, pid 2 2024-04-12T22:03:12.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: pidfile_write: ignore empty --pid-file 2024-04-12T22:03:12.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: load: jerasure load: lrc 2024-04-12T22:03:12.495 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: RocksDB version: 7.9.2 2024-04-12T22:03:12.495 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Git sha 0 2024-04-12T22:03:12.495 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Compile date 2024-04-09 21:18:54 2024-04-12T22:03:12.495 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: DB SUMMARY 2024-04-12T22:03:12.495 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: DB Session ID: 8H8ARK48G3GOARNO23FQ 2024-04-12T22:03:12.495 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: CURRENT file: CURRENT 2024-04-12T22:03:12.495 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: IDENTITY file: IDENTITY 2024-04-12T22:03:12.495 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: MANIFEST file: MANIFEST-000009 size: 1092 Bytes 2024-04-12T22:03:12.495 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 1, files: 000027.sst 2024-04-12T22:03:12.495 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000025.log size: 1040053 ; 2024-04-12T22:03:12.495 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.error_if_exists: 0 2024-04-12T22:03:12.495 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.create_if_missing: 0 2024-04-12T22:03:12.495 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.paranoid_checks: 1 2024-04-12T22:03:12.496 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-12T22:03:12.496 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-12T22:03:12.496 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-12T22:03:12.496 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.env: 0x56213f114c60 2024-04-12T22:03:12.496 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.fs: PosixFileSystem 2024-04-12T22:03:12.496 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.info_log: 0x56214150bae0 2024-04-12T22:03:12.496 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.max_file_opening_threads: 16 2024-04-12T22:03:12.496 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.statistics: (nil) 2024-04-12T22:03:12.497 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.use_fsync: 0 2024-04-12T22:03:12.497 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.max_log_file_size: 0 2024-04-12T22:03:12.497 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-12T22:03:12.497 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-12T22:03:12.498 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.keep_log_file_num: 1000 2024-04-12T22:03:12.498 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.recycle_log_file_num: 0 2024-04-12T22:03:12.498 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.allow_fallocate: 1 2024-04-12T22:03:12.498 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.allow_mmap_reads: 0 2024-04-12T22:03:12.498 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.allow_mmap_writes: 0 2024-04-12T22:03:12.498 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.use_direct_reads: 0 2024-04-12T22:03:12.498 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-12T22:03:12.498 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.create_missing_column_families: 0 2024-04-12T22:03:12.498 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.db_log_dir: 2024-04-12T22:03:12.498 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.wal_dir: 2024-04-12T22:03:12.498 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-12T22:03:12.498 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-12T22:03:12.499 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-12T22:03:12.499 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-12T22:03:12.499 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-12T22:03:12.499 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-12T22:03:12.499 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.advise_random_on_open: 1 2024-04-12T22:03:12.499 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.db_write_buffer_size: 0 2024-04-12T22:03:12.499 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.write_buffer_manager: 0x5621414de820 2024-04-12T22:03:12.499 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-12T22:03:12.499 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-12T22:03:12.499 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-12T22:03:12.499 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.rate_limiter: (nil) 2024-04-12T22:03:12.499 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-12T22:03:12.499 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.wal_recovery_mode: 2 2024-04-12T22:03:12.499 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.enable_thread_tracking: 0 2024-04-12T22:03:12.500 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.enable_pipelined_write: 0 2024-04-12T22:03:12.500 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.unordered_write: 0 2024-04-12T22:03:12.500 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-12T22:03:12.500 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-12T22:03:12.500 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-12T22:03:12.500 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-12T22:03:12.500 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.row_cache: None 2024-04-12T22:03:12.500 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.wal_filter: None 2024-04-12T22:03:12.500 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-12T22:03:12.500 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.allow_ingest_behind: 0 2024-04-12T22:03:12.500 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.two_write_queues: 0 2024-04-12T22:03:12.500 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.manual_wal_flush: 0 2024-04-12T22:03:12.500 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.wal_compression: 0 2024-04-12T22:03:12.500 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.atomic_flush: 0 2024-04-12T22:03:12.500 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-12T22:03:12.501 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-12T22:03:12.501 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-12T22:03:12.501 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.log_readahead_size: 0 2024-04-12T22:03:12.501 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-12T22:03:12.501 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.best_efforts_recovery: 0 2024-04-12T22:03:12.501 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-12T22:03:12.501 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-12T22:03:12.501 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.allow_data_in_errors: 0 2024-04-12T22:03:12.501 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.db_host_id: __hostname__ 2024-04-12T22:03:12.501 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-12T22:03:12.501 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.max_background_jobs: 2 2024-04-12T22:03:12.501 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.max_background_compactions: -1 2024-04-12T22:03:12.501 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.max_subcompactions: 1 2024-04-12T22:03:12.501 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-12T22:03:12.501 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-12T22:03:12.502 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-12T22:03:12.502 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.max_total_wal_size: 0 2024-04-12T22:03:12.502 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-12T22:03:12.502 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-12T22:03:12.502 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-12T22:03:12.502 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-12T22:03:12.502 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.max_open_files: -1 2024-04-12T22:03:12.502 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.bytes_per_sync: 0 2024-04-12T22:03:12.502 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-12T22:03:12.502 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-12T22:03:12.502 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.compaction_readahead_size: 0 2024-04-12T22:03:12.502 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.max_background_flushes: -1 2024-04-12T22:03:12.502 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Compression algorithms supported: 2024-04-12T22:03:12.503 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-12T22:03:12.503 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: kZSTD supported: 0 2024-04-12T22:03:12.503 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: kXpressCompression supported: 0 2024-04-12T22:03:12.503 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: kLZ4HCCompression supported: 1 2024-04-12T22:03:12.503 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: kZlibCompression supported: 1 2024-04-12T22:03:12.503 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: kSnappyCompression supported: 1 2024-04-12T22:03:12.503 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: kLZ4Compression supported: 1 2024-04-12T22:03:12.503 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: kBZip2Compression supported: 0 2024-04-12T22:03:12.503 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-12T22:03:12.503 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-12T22:03:12.503 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 2024-04-12T22:03:12.503 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-12T22:03:12.503 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-12T22:03:12.503 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.merge_operator: 2024-04-12T22:03:12.503 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.compaction_filter: None 2024-04-12T22:03:12.504 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.compaction_filter_factory: None 2024-04-12T22:03:12.504 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.sst_partitioner_factory: None 2024-04-12T22:03:12.504 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-12T22:03:12.504 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-12T22:03:12.504 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56214150bbe0) 2024-04-12T22:03:12.504 INFO:journalctl@ceph.mon.b.smithi114.stdout: cache_index_and_filter_blocks: 1 2024-04-12T22:03:12.504 INFO:journalctl@ceph.mon.b.smithi114.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-12T22:03:12.504 INFO:journalctl@ceph.mon.b.smithi114.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-12T22:03:12.504 INFO:journalctl@ceph.mon.b.smithi114.stdout: pin_top_level_index_and_filter: 1 2024-04-12T22:03:12.504 INFO:journalctl@ceph.mon.b.smithi114.stdout: index_type: 0 2024-04-12T22:03:12.504 INFO:journalctl@ceph.mon.b.smithi114.stdout: data_block_index_type: 0 2024-04-12T22:03:12.504 INFO:journalctl@ceph.mon.b.smithi114.stdout: index_shortening: 1 2024-04-12T22:03:12.504 INFO:journalctl@ceph.mon.b.smithi114.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-12T22:03:12.504 INFO:journalctl@ceph.mon.b.smithi114.stdout: checksum: 4 2024-04-12T22:03:12.505 INFO:journalctl@ceph.mon.b.smithi114.stdout: no_block_cache: 0 2024-04-12T22:03:12.505 INFO:journalctl@ceph.mon.b.smithi114.stdout: block_cache: 0x56214150f090 2024-04-12T22:03:12.505 INFO:journalctl@ceph.mon.b.smithi114.stdout: block_cache_name: BinnedLRUCache 2024-04-12T22:03:12.505 INFO:journalctl@ceph.mon.b.smithi114.stdout: block_cache_options: 2024-04-12T22:03:12.505 INFO:journalctl@ceph.mon.b.smithi114.stdout: capacity : 536870912 2024-04-12T22:03:12.505 INFO:journalctl@ceph.mon.b.smithi114.stdout: num_shard_bits : 4 2024-04-12T22:03:12.505 INFO:journalctl@ceph.mon.b.smithi114.stdout: strict_capacity_limit : 0 2024-04-12T22:03:12.505 INFO:journalctl@ceph.mon.b.smithi114.stdout: high_pri_pool_ratio: 0.000 2024-04-12T22:03:12.505 INFO:journalctl@ceph.mon.b.smithi114.stdout: block_cache_compressed: (nil) 2024-04-12T22:03:12.505 INFO:journalctl@ceph.mon.b.smithi114.stdout: persistent_cache: (nil) 2024-04-12T22:03:12.505 INFO:journalctl@ceph.mon.b.smithi114.stdout: block_size: 4096 2024-04-12T22:03:12.505 INFO:journalctl@ceph.mon.b.smithi114.stdout: block_size_deviation: 10 2024-04-12T22:03:12.505 INFO:journalctl@ceph.mon.b.smithi114.stdout: block_restart_interval: 16 2024-04-12T22:03:12.505 INFO:journalctl@ceph.mon.b.smithi114.stdout: index_block_restart_interval: 1 2024-04-12T22:03:12.505 INFO:journalctl@ceph.mon.b.smithi114.stdout: metadata_block_size: 4096 2024-04-12T22:03:12.506 INFO:journalctl@ceph.mon.b.smithi114.stdout: partition_filters: 0 2024-04-12T22:03:12.506 INFO:journalctl@ceph.mon.b.smithi114.stdout: use_delta_encoding: 1 2024-04-12T22:03:12.506 INFO:journalctl@ceph.mon.b.smithi114.stdout: filter_policy: bloomfilter 2024-04-12T22:03:12.506 INFO:journalctl@ceph.mon.b.smithi114.stdout: whole_key_filtering: 1 2024-04-12T22:03:12.506 INFO:journalctl@ceph.mon.b.smithi114.stdout: verify_compression: 0 2024-04-12T22:03:12.506 INFO:journalctl@ceph.mon.b.smithi114.stdout: read_amp_bytes_per_bit: 0 2024-04-12T22:03:12.506 INFO:journalctl@ceph.mon.b.smithi114.stdout: format_version: 5 2024-04-12T22:03:12.506 INFO:journalctl@ceph.mon.b.smithi114.stdout: enable_index_compression: 1 2024-04-12T22:03:12.506 INFO:journalctl@ceph.mon.b.smithi114.stdout: block_align: 0 2024-04-12T22:03:12.506 INFO:journalctl@ceph.mon.b.smithi114.stdout: max_auto_readahead_size: 262144 2024-04-12T22:03:12.506 INFO:journalctl@ceph.mon.b.smithi114.stdout: prepopulate_block_cache: 0 2024-04-12T22:03:12.506 INFO:journalctl@ceph.mon.b.smithi114.stdout: initial_auto_readahead_size: 8192 2024-04-12T22:03:12.506 INFO:journalctl@ceph.mon.b.smithi114.stdout: num_file_reads_for_auto_readahead: 2 2024-04-12T22:03:12.506 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.write_buffer_size: 33554432 2024-04-12T22:03:12.507 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.max_write_buffer_number: 2 2024-04-12T22:03:12.507 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.compression: NoCompression 2024-04-12T22:03:12.507 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.bottommost_compression: Disabled 2024-04-12T22:03:12.507 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.prefix_extractor: nullptr 2024-04-12T22:03:12.507 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-12T22:03:12.507 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.num_levels: 7 2024-04-12T22:03:12.507 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-12T22:03:12.507 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-12T22:03:12.507 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-12T22:03:12.507 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-12T22:03:12.507 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-12T22:03:12.507 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-12T22:03:12.507 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-12T22:03:12.507 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-12T22:03:12.508 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-12T22:03:12.508 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-12T22:03:12.508 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-12T22:03:12.508 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-12T22:03:12.508 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-12T22:03:12.508 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.compression_opts.level: 32767 2024-04-12T22:03:12.508 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.compression_opts.strategy: 0 2024-04-12T22:03:12.508 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-12T22:03:12.508 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-12T22:03:12.508 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-12T22:03:12.508 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-12T22:03:12.508 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.compression_opts.enabled: false 2024-04-12T22:03:12.508 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-12T22:03:12.508 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-12T22:03:12.508 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-12T22:03:12.509 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-12T22:03:12.509 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.target_file_size_base: 67108864 2024-04-12T22:03:12.509 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-12T22:03:12.509 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-12T22:03:12.509 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-12T22:03:12.509 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-12T22:03:12.509 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-12T22:03:12.509 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-12T22:03:12.509 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-12T22:03:12.509 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-12T22:03:12.509 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-12T22:03:12.509 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-12T22:03:12.509 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-12T22:03:12.509 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-12T22:03:12.510 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-12T22:03:12.510 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-12T22:03:12.510 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.arena_block_size: 1048576 2024-04-12T22:03:12.510 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-12T22:03:12.510 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-12T22:03:12.510 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.disable_auto_compactions: 0 2024-04-12T22:03:12.510 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-12T22:03:12.510 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-12T22:03:12.510 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-12T22:03:12.510 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-12T22:03:12.510 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-12T22:03:12.510 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-12T22:03:12.510 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-12T22:03:12.510 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-12T22:03:12.511 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-12T22:03:12.511 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-12T22:03:12.511 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.table_properties_collectors: 2024-04-12T22:03:12.511 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.inplace_update_support: 0 2024-04-12T22:03:12.511 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-12T22:03:12.511 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-12T22:03:12.511 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-12T22:03:12.511 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-12T22:03:12.511 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.bloom_locality: 0 2024-04-12T22:03:12.511 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.max_successive_merges: 0 2024-04-12T22:03:12.511 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-12T22:03:12.511 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.paranoid_file_checks: 0 2024-04-12T22:03:12.511 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.force_consistency_checks: 1 2024-04-12T22:03:12.511 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.report_bg_io_stats: 0 2024-04-12T22:03:12.511 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.ttl: 2592000 2024-04-12T22:03:12.512 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-12T22:03:12.512 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-12T22:03:12.512 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-12T22:03:12.512 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.enable_blob_files: false 2024-04-12T22:03:12.513 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.min_blob_size: 0 2024-04-12T22:03:12.513 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.blob_file_size: 268435456 2024-04-12T22:03:12.513 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-12T22:03:12.513 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-12T22:03:12.513 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-12T22:03:12.513 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-12T22:03:12.513 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-12T22:03:12.513 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.blob_file_starting_level: 0 2024-04-12T22:03:12.513 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-12T22:03:12.513 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 27.sst 2024-04-12T22:03:12.513 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-12T22:03:12.513 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 29, last_sequence is 15089, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-12T22:03:12.513 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-04-12T22:03:12.514 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 9d0c13eb-c1e3-4135-920f-317c9a8c7906 2024-04-12T22:03:12.514 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712959392096514, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-04-12T22:03:12.514 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-04-12T22:03:12.514 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712959392100848, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 622110, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 15094, "largest_seqno": 15750, "table_properties": {"data_size": 618707, "index_size": 1236, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1157, "raw_key_size": 8956, "raw_average_key_size": 20, "raw_value_size": 611239, "raw_average_value_size": 1389, "num_data_blocks": 55, "num_entries": 440, "num_filter_entries": 440, "num_deletions": 251, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1712959392, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9d0c13eb-c1e3-4135-920f-317c9a8c7906", "db_session_id": "8H8ARK48G3GOARNO23FQ", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-04-12T22:03:12.514 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712959392100954, "job": 1, "event": "recovery_finished"} 2024-04-12T22:03:12.514 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-04-12T22:03:12.514 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-12T22:03:12.514 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-12T22:03:12.514 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x562141600000 2024-04-12T22:03:12.514 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: DB pointer 0x5621415ea000 2024-04-12T22:03:12.514 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-12T22:03:12.514 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-12T22:03:12.514 INFO:journalctl@ceph.mon.b.smithi114.stdout: ** DB Stats ** 2024-04-12T22:03:12.514 INFO:journalctl@ceph.mon.b.smithi114.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-12T22:03:12.515 INFO:journalctl@ceph.mon.b.smithi114.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-12T22:03:12.515 INFO:journalctl@ceph.mon.b.smithi114.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-12T22:03:12.515 INFO:journalctl@ceph.mon.b.smithi114.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-12T22:03:12.515 INFO:journalctl@ceph.mon.b.smithi114.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-12T22:03:12.515 INFO:journalctl@ceph.mon.b.smithi114.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-12T22:03:12.515 INFO:journalctl@ceph.mon.b.smithi114.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-12T22:03:12.515 INFO:journalctl@ceph.mon.b.smithi114.stdout: 2024-04-12T22:03:12.515 INFO:journalctl@ceph.mon.b.smithi114.stdout: ** Compaction Stats [default] ** 2024-04-12T22:03:12.515 INFO:journalctl@ceph.mon.b.smithi114.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-12T22:03:12.515 INFO:journalctl@ceph.mon.b.smithi114.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-12T22:03:12.515 INFO:journalctl@ceph.mon.b.smithi114.stdout: L0 1/0 607.53 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 258.2 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-04-12T22:03:12.515 INFO:journalctl@ceph.mon.b.smithi114.stdout: L6 1/0 11.56 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-12T22:03:12.515 INFO:journalctl@ceph.mon.b.smithi114.stdout: Sum 2/0 12.15 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 258.2 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-04-12T22:03:12.515 INFO:journalctl@ceph.mon.b.smithi114.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 258.2 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-04-12T22:03:12.516 INFO:journalctl@ceph.mon.b.smithi114.stdout: 2024-04-12T22:03:12.516 INFO:journalctl@ceph.mon.b.smithi114.stdout: ** Compaction Stats [default] ** 2024-04-12T22:03:12.516 INFO:journalctl@ceph.mon.b.smithi114.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-12T22:03:12.516 INFO:journalctl@ceph.mon.b.smithi114.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-12T22:03:12.516 INFO:journalctl@ceph.mon.b.smithi114.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 258.2 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-04-12T22:03:12.516 INFO:journalctl@ceph.mon.b.smithi114.stdout: 2024-04-12T22:03:12.516 INFO:journalctl@ceph.mon.b.smithi114.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-12T22:03:12.516 INFO:journalctl@ceph.mon.b.smithi114.stdout: 2024-04-12T22:03:12.516 INFO:journalctl@ceph.mon.b.smithi114.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-12T22:03:12.516 INFO:journalctl@ceph.mon.b.smithi114.stdout: Flush(GB): cumulative 0.001, interval 0.001 2024-04-12T22:03:12.516 INFO:journalctl@ceph.mon.b.smithi114.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-12T22:03:12.516 INFO:journalctl@ceph.mon.b.smithi114.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-12T22:03:12.516 INFO:journalctl@ceph.mon.b.smithi114.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-12T22:03:12.516 INFO:journalctl@ceph.mon.b.smithi114.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-12T22:03:12.516 INFO:journalctl@ceph.mon.b.smithi114.stdout: Cumulative compaction: 0.00 GB write, 85.22 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-12T22:03:12.517 INFO:journalctl@ceph.mon.b.smithi114.stdout: Interval compaction: 0.00 GB write, 85.22 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-12T22:03:12.517 INFO:journalctl@ceph.mon.b.smithi114.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-12T22:03:12.517 INFO:journalctl@ceph.mon.b.smithi114.stdout: Block cache BinnedLRUCache@0x56214150f090#2 capacity: 512.00 MB usage: 2.61 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-04-12T22:03:12.517 INFO:journalctl@ceph.mon.b.smithi114.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.28 KB,0.000244379%) IndexBlock(1,1.33 KB,0.00025332%) Misc(1,0.00 KB,0%) 2024-04-12T22:03:12.517 INFO:journalctl@ceph.mon.b.smithi114.stdout: 2024-04-12T22:03:12.517 INFO:journalctl@ceph.mon.b.smithi114.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-12T22:03:12.517 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: starting mon.b rank 2 at public addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] at bind addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 80bf7680-f916-11ee-bc8d-c7b262605968 2024-04-12T22:03:12.517 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: mon.b@-1(???) e3 preinit fsid 80bf7680-f916-11ee-bc8d-c7b262605968 2024-04-12T22:03:12.517 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: mon.b@-1(???).mds e1 new map 2024-04-12T22:03:12.517 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: mon.b@-1(???).mds e1 print_map 2024-04-12T22:03:12.517 INFO:journalctl@ceph.mon.b.smithi114.stdout: e1 2024-04-12T22:03:12.517 INFO:journalctl@ceph.mon.b.smithi114.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T22:03:12.517 INFO:journalctl@ceph.mon.b.smithi114.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-12T22:03:12.517 INFO:journalctl@ceph.mon.b.smithi114.stdout: legacy client fscid: -1 2024-04-12T22:03:12.518 INFO:journalctl@ceph.mon.b.smithi114.stdout: 2024-04-12T22:03:12.518 INFO:journalctl@ceph.mon.b.smithi114.stdout: No filesystems configured 2024-04-12T22:03:12.518 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: mon.b@-1(???).osd e106 crush map has features 3314933000854323200, adjusting msgr requires 2024-04-12T22:03:12.518 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: mon.b@-1(???).osd e106 crush map has features 432629239337189376, adjusting msgr requires 2024-04-12T22:03:12.518 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: mon.b@-1(???).osd e106 crush map has features 432629239337189376, adjusting msgr requires 2024-04-12T22:03:12.518 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: mon.b@-1(???).osd e106 crush map has features 432629239337189376, adjusting msgr requires 2024-04-12T22:03:12.518 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 ceph-mon[58780]: mon.b@-1(???).paxosservice(auth 1..24) refresh upgraded, format 0 -> 3 2024-04-12T22:03:12.518 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 bash[58761]: 71a2eff159a33616171e9ceaec3ab3a21f9f32db7fe3fb91524b7097f74f0d55 2024-04-12T22:03:12.518 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:12 smithi114 systemd[1]: Started Ceph mon.b for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:03:12.941 INFO:teuthology.orchestra.run.smithi097.stdout:true 2024-04-12T22:03:17.747 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:17 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@grafana.a.service: Scheduled restart job, restart counter is at 9. 2024-04-12T22:03:17.747 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:17 smithi114 systemd[1]: Stopped Ceph grafana.a for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:03:17.747 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:17 smithi114 systemd[1]: Starting Ceph grafana.a for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:03:17.747 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:17 smithi114 podman[58874]: 2024-04-12 22:03:17.63405739 +0000 UTC m=+0.017918509 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-04-12T22:03:18.178 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:17 smithi114 podman[58874]: 2024-04-12 22:03:17.746393021 +0000 UTC m=+0.130254125 container create e428ee72861aac751b347bf7986518de7428ba1bb22df8d5194def09d529cdbe (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, com.redhat.component=ubi8-container, architecture=x86_64, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, summary=Grafana Container configured for Ceph mgr/dashboard integration, name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.buildah.version=1.24.2, maintainer=Paul Cuzner , release=236.1648460182, version=8.5, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container, io.k8s.display-name=Red Hat Universal Base Image 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-type=git, distribution-scope=public, build-date=2022-03-28T10:36:18.413762, vendor=Red Hat, Inc., io.openshift.tags=base rhel8, io.openshift.expose-services=) 2024-04-12T22:03:18.661 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:18 smithi114 podman[58874]: 2024-04-12 22:03:18.315574761 +0000 UTC m=+0.699435865 container init e428ee72861aac751b347bf7986518de7428ba1bb22df8d5194def09d529cdbe (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, build-date=2022-03-28T10:36:18.413762, name=ubi8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., version=8.5, io.openshift.tags=base rhel8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, description=Ceph Grafana Container, io.openshift.expose-services=, com.redhat.component=ubi8-container, vcs-type=git, summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, io.k8s.display-name=Red Hat Universal Base Image 8, io.buildah.version=1.24.2, maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, release=236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI) 2024-04-12T22:03:18.661 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:18 smithi114 podman[58874]: 2024-04-12 22:03:18.318287066 +0000 UTC m=+0.702148173 container start e428ee72861aac751b347bf7986518de7428ba1bb22df8d5194def09d529cdbe (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, build-date=2022-03-28T10:36:18.413762, name=ubi8, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, distribution-scope=public, io.openshift.tags=base rhel8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=ubi8-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8.5, maintainer=Paul Cuzner , release=236.1648460182, vcs-type=git, summary=Grafana Container configured for Ceph mgr/dashboard integration, description=Ceph Grafana Container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.24.2, io.openshift.expose-services=, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI) 2024-04-12T22:03:18.661 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:18 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[58955]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-12T22:03:18.661 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:18 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[58955]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-12T22:03:18.661 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:18 smithi114 bash[58874]: e428ee72861aac751b347bf7986518de7428ba1bb22df8d5194def09d529cdbe 2024-04-12T22:03:18.661 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:18 smithi114 systemd[1]: Started Ceph grafana.a for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:03:18.661 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:18 smithi114 podman[58997]: 2024-04-12 22:03:18.406619821 +0000 UTC m=+0.052561385 container died e428ee72861aac751b347bf7986518de7428ba1bb22df8d5194def09d529cdbe (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, io.openshift.tags=base rhel8, description=Ceph Grafana Container, io.openshift.expose-services=, io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8, build-date=2022-03-28T10:36:18.413762, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, summary=Grafana Container configured for Ceph mgr/dashboard integration, version=8.5, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.component=ubi8-container, release=236.1648460182, vcs-type=git, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Paul Cuzner , io.buildah.version=1.24.2, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc.) 2024-04-12T22:03:18.944 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:03:18 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:03:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"smithi097:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"172.21.15.114:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-12T22:03:19.243 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:18 smithi114 podman[58997]: 2024-04-12 22:03:18.943922904 +0000 UTC m=+0.589864463 container remove e428ee72861aac751b347bf7986518de7428ba1bb22df8d5194def09d529cdbe (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, io.buildah.version=1.24.2, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, release=236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, com.redhat.component=ubi8-container, version=8.5, summary=Grafana Container configured for Ceph mgr/dashboard integration, build-date=2022-03-28T10:36:18.413762, distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.openshift.tags=base rhel8, maintainer=Paul Cuzner , architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=ubi8, vcs-type=git, vendor=Red Hat, Inc., description=Ceph Grafana Container) 2024-04-12T22:03:19.243 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:18 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-04-12T22:03:19.243 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:18 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-04-12T22:03:19.660 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:03:19 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:03:19] "GET /metrics HTTP/1.1" 200 38264 "" "Prometheus/2.33.4" 2024-04-12T22:03:22.410 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:03:22 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:03:22.111+0000 7fe40fa5a700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-12T22:03:25.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[67407]: mon.c calling monitor election 2024-04-12T22:03:25.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[67407]: mon.b calling monitor election 2024-04-12T22:03:25.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[67407]: mon.c calling monitor election 2024-04-12T22:03:25.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[67407]: monmap epoch 4 2024-04-12T22:03:25.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[67407]: fsid 80bf7680-f916-11ee-bc8d-c7b262605968 2024-04-12T22:03:25.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[67407]: last_changed 2024-04-12T22:03:18.724970+0000 2024-04-12T22:03:25.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[67407]: created 2024-04-12T21:49:45.534140+0000 2024-04-12T22:03:25.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[67407]: min_mon_release 19 (squid) 2024-04-12T22:03:25.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[67407]: election_strategy: 1 2024-04-12T22:03:25.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[67407]: 0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2024-04-12T22:03:25.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[67407]: 1: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2024-04-12T22:03:25.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[67407]: 2: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-04-12T22:03:25.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[67407]: fsmap 2024-04-12T22:03:25.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[67407]: osdmap e106: 8 total, 8 up, 8 in 2024-04-12T22:03:25.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[67407]: mgrmap e38: y(active, since 44s), standbys: x 2024-04-12T22:03:25.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[67407]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-12T22:03:25.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[67407]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-12T22:03:25.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[67407]: daemon grafana.a on smithi114 is in unknown state 2024-04-12T22:03:25.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[67407]: mon.a calling monitor election 2024-04-12T22:03:25.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[67407]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-12T22:03:25.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[67407]: monmap epoch 4 2024-04-12T22:03:25.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[67407]: fsid 80bf7680-f916-11ee-bc8d-c7b262605968 2024-04-12T22:03:25.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[67407]: last_changed 2024-04-12T22:03:18.724970+0000 2024-04-12T22:03:25.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[67407]: created 2024-04-12T21:49:45.534140+0000 2024-04-12T22:03:25.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[67407]: min_mon_release 19 (squid) 2024-04-12T22:03:25.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[67407]: election_strategy: 1 2024-04-12T22:03:25.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[67407]: 0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2024-04-12T22:03:25.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[67407]: 1: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2024-04-12T22:03:25.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[67407]: 2: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-04-12T22:03:25.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[67407]: fsmap 2024-04-12T22:03:25.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[67407]: osdmap e106: 8 total, 8 up, 8 in 2024-04-12T22:03:25.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[67407]: mgrmap e38: y(active, since 44s), standbys: x 2024-04-12T22:03:25.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[67407]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-12T22:03:25.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[67407]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-12T22:03:25.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[67407]: daemon grafana.a on smithi114 is in unknown state 2024-04-12T22:03:25.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:25.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:25.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[64853]: mon.c calling monitor election 2024-04-12T22:03:25.163 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[64853]: mon.b calling monitor election 2024-04-12T22:03:25.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[64853]: mon.c calling monitor election 2024-04-12T22:03:25.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[64853]: monmap epoch 4 2024-04-12T22:03:25.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[64853]: fsid 80bf7680-f916-11ee-bc8d-c7b262605968 2024-04-12T22:03:25.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[64853]: last_changed 2024-04-12T22:03:18.724970+0000 2024-04-12T22:03:25.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[64853]: created 2024-04-12T21:49:45.534140+0000 2024-04-12T22:03:25.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[64853]: min_mon_release 19 (squid) 2024-04-12T22:03:25.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[64853]: election_strategy: 1 2024-04-12T22:03:25.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[64853]: 0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2024-04-12T22:03:25.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[64853]: 1: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2024-04-12T22:03:25.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[64853]: 2: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-04-12T22:03:25.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[64853]: fsmap 2024-04-12T22:03:25.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[64853]: osdmap e106: 8 total, 8 up, 8 in 2024-04-12T22:03:25.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[64853]: mgrmap e38: y(active, since 44s), standbys: x 2024-04-12T22:03:25.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[64853]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-12T22:03:25.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[64853]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-12T22:03:25.164 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[64853]: daemon grafana.a on smithi114 is in unknown state 2024-04-12T22:03:25.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[64853]: mon.a calling monitor election 2024-04-12T22:03:25.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[64853]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-12T22:03:25.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[64853]: monmap epoch 4 2024-04-12T22:03:25.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[64853]: fsid 80bf7680-f916-11ee-bc8d-c7b262605968 2024-04-12T22:03:25.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[64853]: last_changed 2024-04-12T22:03:18.724970+0000 2024-04-12T22:03:25.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[64853]: created 2024-04-12T21:49:45.534140+0000 2024-04-12T22:03:25.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[64853]: min_mon_release 19 (squid) 2024-04-12T22:03:25.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[64853]: election_strategy: 1 2024-04-12T22:03:25.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[64853]: 0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2024-04-12T22:03:25.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[64853]: 1: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2024-04-12T22:03:25.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[64853]: 2: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-04-12T22:03:25.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[64853]: fsmap 2024-04-12T22:03:25.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[64853]: osdmap e106: 8 total, 8 up, 8 in 2024-04-12T22:03:25.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[64853]: mgrmap e38: y(active, since 44s), standbys: x 2024-04-12T22:03:25.165 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[64853]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-12T22:03:25.166 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[64853]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-12T22:03:25.166 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[64853]: daemon grafana.a on smithi114 is in unknown state 2024-04-12T22:03:25.166 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:25.166 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:24 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:25.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:24 smithi114 ceph-mon[58780]: mon.c calling monitor election 2024-04-12T22:03:25.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:24 smithi114 ceph-mon[58780]: mon.b calling monitor election 2024-04-12T22:03:25.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:24 smithi114 ceph-mon[58780]: mon.c calling monitor election 2024-04-12T22:03:25.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:24 smithi114 ceph-mon[58780]: monmap epoch 4 2024-04-12T22:03:25.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:24 smithi114 ceph-mon[58780]: fsid 80bf7680-f916-11ee-bc8d-c7b262605968 2024-04-12T22:03:25.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:24 smithi114 ceph-mon[58780]: last_changed 2024-04-12T22:03:18.724970+0000 2024-04-12T22:03:25.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:24 smithi114 ceph-mon[58780]: created 2024-04-12T21:49:45.534140+0000 2024-04-12T22:03:25.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:24 smithi114 ceph-mon[58780]: min_mon_release 19 (squid) 2024-04-12T22:03:25.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:24 smithi114 ceph-mon[58780]: election_strategy: 1 2024-04-12T22:03:25.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:24 smithi114 ceph-mon[58780]: 0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2024-04-12T22:03:25.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:24 smithi114 ceph-mon[58780]: 1: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2024-04-12T22:03:25.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:24 smithi114 ceph-mon[58780]: 2: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-04-12T22:03:25.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:24 smithi114 ceph-mon[58780]: fsmap 2024-04-12T22:03:25.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:24 smithi114 ceph-mon[58780]: osdmap e106: 8 total, 8 up, 8 in 2024-04-12T22:03:25.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:24 smithi114 ceph-mon[58780]: mgrmap e38: y(active, since 44s), standbys: x 2024-04-12T22:03:25.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:24 smithi114 ceph-mon[58780]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-12T22:03:25.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:24 smithi114 ceph-mon[58780]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-12T22:03:25.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:24 smithi114 ceph-mon[58780]: daemon grafana.a on smithi114 is in unknown state 2024-04-12T22:03:25.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:24 smithi114 ceph-mon[58780]: mon.a calling monitor election 2024-04-12T22:03:25.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:24 smithi114 ceph-mon[58780]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-12T22:03:25.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:24 smithi114 ceph-mon[58780]: monmap epoch 4 2024-04-12T22:03:25.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:24 smithi114 ceph-mon[58780]: fsid 80bf7680-f916-11ee-bc8d-c7b262605968 2024-04-12T22:03:25.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:24 smithi114 ceph-mon[58780]: last_changed 2024-04-12T22:03:18.724970+0000 2024-04-12T22:03:25.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:24 smithi114 ceph-mon[58780]: created 2024-04-12T21:49:45.534140+0000 2024-04-12T22:03:25.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:24 smithi114 ceph-mon[58780]: min_mon_release 19 (squid) 2024-04-12T22:03:25.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:24 smithi114 ceph-mon[58780]: election_strategy: 1 2024-04-12T22:03:25.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:24 smithi114 ceph-mon[58780]: 0: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.a 2024-04-12T22:03:25.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:24 smithi114 ceph-mon[58780]: 1: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2024-04-12T22:03:25.246 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:24 smithi114 ceph-mon[58780]: 2: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.b 2024-04-12T22:03:25.246 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:24 smithi114 ceph-mon[58780]: fsmap 2024-04-12T22:03:25.246 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:24 smithi114 ceph-mon[58780]: osdmap e106: 8 total, 8 up, 8 in 2024-04-12T22:03:25.246 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:24 smithi114 ceph-mon[58780]: mgrmap e38: y(active, since 44s), standbys: x 2024-04-12T22:03:25.246 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:24 smithi114 ceph-mon[58780]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-12T22:03:25.246 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:24 smithi114 ceph-mon[58780]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-12T22:03:25.246 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:24 smithi114 ceph-mon[58780]: daemon grafana.a on smithi114 is in unknown state 2024-04-12T22:03:25.246 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:24 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:25.246 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:24 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:25.593 INFO:teuthology.orchestra.run.smithi097.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T22:03:25.593 INFO:teuthology.orchestra.run.smithi097.stdout:alertmanager.a smithi097 *:9093,9094 running (2m) 23s ago 7m 17.8M - 0.23.0 ba2b418f427c 7b4c491fe0a0 2024-04-12T22:03:25.594 INFO:teuthology.orchestra.run.smithi097.stdout:grafana.a smithi114 *:3000 unknown 3s ago 7m - - 2024-04-12T22:03:25.594 INFO:teuthology.orchestra.run.smithi097.stdout:iscsi.foo.smithi097.xndkxy smithi097 running (81s) 23s ago 6m 47.4M - 3.5 e1d6a67b021e 0436463f28cd 2024-04-12T22:03:25.594 INFO:teuthology.orchestra.run.smithi097.stdout:mgr.x smithi114 *:8443,9283,8765 running (71s) 3s ago 11m 450M - 19.0.0-2053-gb277feb2 668895944f0e b8f8df3bf928 2024-04-12T22:03:25.594 INFO:teuthology.orchestra.run.smithi097.stdout:mgr.y smithi097 *:8443,9283,8765 running (107s) 23s ago 13m 494M - 19.0.0-2053-gb277feb2 668895944f0e 488170dac11f 2024-04-12T22:03:25.594 INFO:teuthology.orchestra.run.smithi097.stdout:mon.a smithi097 running (56s) 23s ago 13m 41.1M 2048M 19.0.0-2053-gb277feb2 668895944f0e 3cfa16d3a85c 2024-04-12T22:03:25.594 INFO:teuthology.orchestra.run.smithi097.stdout:mon.b smithi114 running (13s) 3s ago 12m 18.4M 2048M 19.0.0-2053-gb277feb2 668895944f0e 71a2eff159a3 2024-04-12T22:03:25.594 INFO:teuthology.orchestra.run.smithi097.stdout:mon.c smithi097 running (30s) 23s ago 12m 18.0M 2048M 19.0.0-2053-gb277feb2 668895944f0e 69507ba88603 2024-04-12T22:03:25.594 INFO:teuthology.orchestra.run.smithi097.stdout:node-exporter.a smithi097 *:9100 running (2m) 23s ago 8m 19.2M - 1.3.1 1dbe0e931976 5edff9053b37 2024-04-12T22:03:25.594 INFO:teuthology.orchestra.run.smithi097.stdout:node-exporter.b smithi114 *:9100 running (2m) 3s ago 7m 17.0M - 1.3.1 1dbe0e931976 904afc45dc2f 2024-04-12T22:03:25.594 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0 smithi097 running (11m) 23s ago 11m 55.0M 2736M 17.2.0 e1d6a67b021e abac1d8884fa 2024-04-12T22:03:25.594 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1 smithi097 running (11m) 23s ago 11m 54.3M 2736M 17.2.0 e1d6a67b021e 39192562d598 2024-04-12T22:03:25.594 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2 smithi097 running (10m) 23s ago 10m 51.3M 2736M 17.2.0 e1d6a67b021e 89daf8e339c3 2024-04-12T22:03:25.594 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3 smithi097 running (10m) 23s ago 10m 51.8M 2736M 17.2.0 e1d6a67b021e 4e9a0b3862d9 2024-04-12T22:03:25.595 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4 smithi114 running (9m) 3s ago 9m 53.6M 2992M 17.2.0 e1d6a67b021e fb42c0e8e575 2024-04-12T22:03:25.595 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5 smithi114 running (9m) 3s ago 9m 50.2M 2992M 17.2.0 e1d6a67b021e 3acd4a3b61e5 2024-04-12T22:03:25.595 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6 smithi114 running (9m) 3s ago 9m 49.6M 2992M 17.2.0 e1d6a67b021e 5969baf67f41 2024-04-12T22:03:25.595 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7 smithi114 running (8m) 3s ago 8m 51.8M 2992M 17.2.0 e1d6a67b021e f68340fecf2c 2024-04-12T22:03:25.595 INFO:teuthology.orchestra.run.smithi097.stdout:prometheus.a smithi114 *:9095 running (77s) 3s ago 8m 65.1M - 2.33.4 514e6a882f6e 0ad5437ed63f 2024-04-12T22:03:25.595 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.foo.smithi097.sggtkh smithi097 *:8000 running (6m) 23s ago 6m 72.2M - 17.2.0 e1d6a67b021e a39f9b2d8509 2024-04-12T22:03:25.595 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.foo.smithi114.ulfbyo smithi114 *:8000 running (6m) 3s ago 6m 72.8M - 17.2.0 e1d6a67b021e df79a48bdd59 2024-04-12T22:03:25.595 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.smpl.smithi097.exxkgl smithi097 *:80 running (6m) 23s ago 6m 72.2M - 17.2.0 e1d6a67b021e a036c58a0099 2024-04-12T22:03:25.595 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.smpl.smithi114.lrgvph smithi114 *:80 running (6m) 3s ago 6m 74.4M - 17.2.0 e1d6a67b021e 031a5fc24237 2024-04-12T22:03:25.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:25 smithi097 ceph-mon[64853]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:03:25.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:25 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:03:25.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:25 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:25.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:25 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:25.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:25 smithi097 ceph-mon[67407]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:03:25.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:25 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:03:25.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:25 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:25.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:25 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:25.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:25 smithi114 ceph-mon[58780]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:03:25.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:25 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:03:25.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:25 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:25.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:25 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:25.997 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-12T22:03:25.997 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": { 2024-04-12T22:03:25.997 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 3 2024-04-12T22:03:25.997 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:03:25.997 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": { 2024-04-12T22:03:25.997 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 2 2024-04-12T22:03:25.997 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:03:25.997 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": { 2024-04-12T22:03:25.997 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-12T22:03:25.997 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:03:25.998 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw": { 2024-04-12T22:03:25.998 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-12T22:03:25.998 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:03:25.998 INFO:teuthology.orchestra.run.smithi097.stdout: "overall": { 2024-04-12T22:03:25.998 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-04-12T22:03:25.998 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 5 2024-04-12T22:03:25.998 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-12T22:03:25.998 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-12T22:03:26.331 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-12T22:03:26.332 INFO:teuthology.orchestra.run.smithi097.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a", 2024-04-12T22:03:26.332 INFO:teuthology.orchestra.run.smithi097.stdout: "in_progress": true, 2024-04-12T22:03:26.332 INFO:teuthology.orchestra.run.smithi097.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T22:03:26.332 INFO:teuthology.orchestra.run.smithi097.stdout: "services_complete": [ 2024-04-12T22:03:26.332 INFO:teuthology.orchestra.run.smithi097.stdout: "mon", 2024-04-12T22:03:26.332 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr" 2024-04-12T22:03:26.332 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-04-12T22:03:26.332 INFO:teuthology.orchestra.run.smithi097.stdout: "progress": "5/23 daemons upgraded", 2024-04-12T22:03:26.332 INFO:teuthology.orchestra.run.smithi097.stdout: "message": "Currently upgrading mon daemons", 2024-04-12T22:03:26.332 INFO:teuthology.orchestra.run.smithi097.stdout: "is_paused": false 2024-04-12T22:03:26.332 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-12T22:03:26.723 INFO:teuthology.orchestra.run.smithi097.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-12T22:03:26.723 INFO:teuthology.orchestra.run.smithi097.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-12T22:03:26.723 INFO:teuthology.orchestra.run.smithi097.stdout: daemon grafana.a on smithi114 is in unknown state 2024-04-12T22:03:26.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:26 smithi114 ceph-mon[58780]: from='client.34176 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:03:26.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:26 smithi114 ceph-mon[58780]: from='client.34182 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:03:26.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:26 smithi114 ceph-mon[58780]: from='client.? 172.21.15.97:0/3459356215' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:03:26.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:26 smithi114 ceph-mon[58780]: from='client.? 172.21.15.97:0/2789766161' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T22:03:27.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:26 smithi097 ceph-mon[64853]: from='client.34176 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:03:27.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:26 smithi097 ceph-mon[64853]: from='client.34182 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:03:27.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:26 smithi097 ceph-mon[64853]: from='client.? 172.21.15.97:0/3459356215' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:03:27.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:26 smithi097 ceph-mon[64853]: from='client.? 172.21.15.97:0/2789766161' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T22:03:27.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:26 smithi097 ceph-mon[67407]: from='client.34176 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:03:27.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:26 smithi097 ceph-mon[67407]: from='client.34182 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:03:27.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:26 smithi097 ceph-mon[67407]: from='client.? 172.21.15.97:0/3459356215' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:03:27.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:26 smithi097 ceph-mon[67407]: from='client.? 172.21.15.97:0/2789766161' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T22:03:28.086 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:27 smithi114 ceph-mon[58780]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:03:28.086 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:27 smithi114 ceph-mon[58780]: from='client.44125 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:03:28.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:27 smithi097 ceph-mon[64853]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:03:28.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:27 smithi097 ceph-mon[64853]: from='client.44125 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:03:28.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:27 smithi097 ceph-mon[67407]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:03:28.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:27 smithi097 ceph-mon[67407]: from='client.44125 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:03:28.993 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:03:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:03:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"smithi097:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"172.21.15.114:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-12T22:03:29.254 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:28 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@grafana.a.service: Scheduled restart job, restart counter is at 10. 2024-04-12T22:03:29.254 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:28 smithi114 systemd[1]: Stopped Ceph grafana.a for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:03:29.255 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:29 smithi114 systemd[1]: Starting Ceph grafana.a for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:03:29.255 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:29 smithi114 podman[59952]: 2024-04-12 22:03:29.138705087 +0000 UTC m=+0.017889088 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-04-12T22:03:29.255 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:29 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:29.255 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:29 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:29.255 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:29 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:29.255 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:29 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:03:29.255 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:29 smithi114 ceph-mon[58780]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 864 B/s rd, 0 op/s 2024-04-12T22:03:29.255 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:29 smithi114 ceph-mon[58780]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-12T22:03:29.255 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:29 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:29.255 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:29 smithi114 ceph-mon[58780]: Reconfiguring mon.a (monmap changed)... 2024-04-12T22:03:29.255 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:29 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T22:03:29.256 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:29 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T22:03:29.256 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:29 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:29.256 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:29 smithi114 ceph-mon[58780]: Reconfiguring daemon mon.a on smithi097 2024-04-12T22:03:29.410 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:03:29 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:03:29] "GET /metrics HTTP/1.1" 200 38275 "" "Prometheus/2.33.4" 2024-04-12T22:03:29.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:29 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:29.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:29 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:29.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:29 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:29.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:29 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:03:29.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:29 smithi097 ceph-mon[67407]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 864 B/s rd, 0 op/s 2024-04-12T22:03:29.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:29 smithi097 ceph-mon[67407]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-12T22:03:29.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:29 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:29.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:29 smithi097 ceph-mon[67407]: Reconfiguring mon.a (monmap changed)... 2024-04-12T22:03:29.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:29 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T22:03:29.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:29 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T22:03:29.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:29 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:29.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:29 smithi097 ceph-mon[67407]: Reconfiguring daemon mon.a on smithi097 2024-04-12T22:03:29.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:29 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:29.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:29 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:29.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:29 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:29.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:29 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:03:29.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:29 smithi097 ceph-mon[64853]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 864 B/s rd, 0 op/s 2024-04-12T22:03:29.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:29 smithi097 ceph-mon[64853]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-12T22:03:29.413 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:29 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:29.413 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:29 smithi097 ceph-mon[64853]: Reconfiguring mon.a (monmap changed)... 2024-04-12T22:03:29.413 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:29 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T22:03:29.413 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:29 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T22:03:29.413 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:29 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:29.413 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:29 smithi097 ceph-mon[64853]: Reconfiguring daemon mon.a on smithi097 2024-04-12T22:03:29.582 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:29 smithi114 podman[59952]: 2024-04-12 22:03:29.254343794 +0000 UTC m=+0.133527781 container create 3db11123d3ae5c86f77844302993a6e709a3b006e0a8a7efed4ecafc78f3128e (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, architecture=x86_64, vcs-type=git, version=8.5, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.expose-services=, io.openshift.tags=base rhel8, maintainer=Paul Cuzner , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=ubi8-container, distribution-scope=public, release=236.1648460182, description=Ceph Grafana Container, io.buildah.version=1.24.2, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, build-date=2022-03-28T10:36:18.413762, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vendor=Red Hat, Inc.) 2024-04-12T22:03:29.582 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:29 smithi114 podman[59952]: 2024-04-12 22:03:29.47339052 +0000 UTC m=+0.352574504 container init 3db11123d3ae5c86f77844302993a6e709a3b006e0a8a7efed4ecafc78f3128e (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, io.openshift.expose-services=, version=8.5, com.redhat.component=ubi8-container, distribution-scope=public, io.openshift.tags=base rhel8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=236.1648460182, maintainer=Paul Cuzner , architecture=x86_64, io.buildah.version=1.24.2, io.k8s.display-name=Red Hat Universal Base Image 8, description=Ceph Grafana Container, build-date=2022-03-28T10:36:18.413762, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vendor=Red Hat, Inc., summary=Grafana Container configured for Ceph mgr/dashboard integration, name=ubi8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182) 2024-04-12T22:03:29.582 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:29 smithi114 podman[59952]: 2024-04-12 22:03:29.476013578 +0000 UTC m=+0.355197572 container start 3db11123d3ae5c86f77844302993a6e709a3b006e0a8a7efed4ecafc78f3128e (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.openshift.expose-services=, io.openshift.tags=base rhel8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=236.1648460182, distribution-scope=public, io.buildah.version=1.24.2, build-date=2022-03-28T10:36:18.413762, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vendor=Red Hat, Inc., description=Ceph Grafana Container, architecture=x86_64, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.component=ubi8-container, name=ubi8, maintainer=Paul Cuzner ) 2024-04-12T22:03:29.582 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:29 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[59968]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-12T22:03:29.582 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:29 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[59968]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-12T22:03:29.993 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:29 smithi114 bash[59952]: 3db11123d3ae5c86f77844302993a6e709a3b006e0a8a7efed4ecafc78f3128e 2024-04-12T22:03:29.993 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:29 smithi114 systemd[1]: Started Ceph grafana.a for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:03:29.993 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:29 smithi114 podman[59984]: 2024-04-12 22:03:29.606303581 +0000 UTC m=+0.096167262 container died 3db11123d3ae5c86f77844302993a6e709a3b006e0a8a7efed4ecafc78f3128e (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., version=8.5, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, release=236.1648460182, build-date=2022-03-28T10:36:18.413762, description=Ceph Grafana Container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, distribution-scope=public, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, maintainer=Paul Cuzner , architecture=x86_64, com.redhat.component=ubi8-container, io.k8s.display-name=Red Hat Universal Base Image 8, io.buildah.version=1.24.2, io.openshift.tags=base rhel8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, name=ubi8) 2024-04-12T22:03:30.493 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:30 smithi114 podman[59984]: 2024-04-12 22:03:30.16029308 +0000 UTC m=+0.650156763 container remove 3db11123d3ae5c86f77844302993a6e709a3b006e0a8a7efed4ecafc78f3128e (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, maintainer=Paul Cuzner , vcs-type=git, io.openshift.tags=base rhel8, vendor=Red Hat, Inc., distribution-scope=public, version=8.5, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-03-28T10:36:18.413762, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.24.2, release=236.1648460182, io.k8s.display-name=Red Hat Universal Base Image 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=ubi8, io.openshift.expose-services=, com.redhat.component=ubi8-container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, description=Ceph Grafana Container, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182) 2024-04-12T22:03:30.493 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:30 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-04-12T22:03:30.494 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:30 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-04-12T22:03:31.159 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:30 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:31.159 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:30 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:31.159 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:30 smithi097 ceph-mon[64853]: Reconfiguring mgr.y (monmap changed)... 2024-04-12T22:03:31.159 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:30 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T22:03:31.159 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:30 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T22:03:31.159 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:30 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:31.159 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:30 smithi097 ceph-mon[64853]: Reconfiguring daemon mgr.y on smithi097 2024-04-12T22:03:31.159 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:30 smithi097 ceph-mon[64853]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-12T22:03:31.159 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:30 smithi097 ceph-mon[64853]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:03:31.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:30 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:31.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:30 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:31.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:30 smithi097 ceph-mon[67407]: Reconfiguring mgr.y (monmap changed)... 2024-04-12T22:03:31.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:30 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T22:03:31.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:30 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T22:03:31.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:30 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:31.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:30 smithi097 ceph-mon[67407]: Reconfiguring daemon mgr.y on smithi097 2024-04-12T22:03:31.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:30 smithi097 ceph-mon[67407]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-12T22:03:31.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:30 smithi097 ceph-mon[67407]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:03:31.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:30 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:31.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:30 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:31.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:30 smithi114 ceph-mon[58780]: Reconfiguring mgr.y (monmap changed)... 2024-04-12T22:03:31.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:30 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T22:03:31.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:30 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T22:03:31.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:30 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:31.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:30 smithi114 ceph-mon[58780]: Reconfiguring daemon mgr.y on smithi097 2024-04-12T22:03:31.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:30 smithi114 ceph-mon[58780]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-12T22:03:31.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:30 smithi114 ceph-mon[58780]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:03:32.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:32 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:32.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:32 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:32.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:32 smithi114 ceph-mon[58780]: Reconfiguring mon.c (monmap changed)... 2024-04-12T22:03:32.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:32 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T22:03:32.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:32 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T22:03:32.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:32 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:32.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:32 smithi114 ceph-mon[58780]: Reconfiguring daemon mon.c on smithi097 2024-04-12T22:03:32.910 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:32 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:32.910 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:32 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:32.910 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:32 smithi097 ceph-mon[67407]: Reconfiguring mon.c (monmap changed)... 2024-04-12T22:03:32.910 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:32 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T22:03:32.910 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:32 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T22:03:32.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:32 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:32.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:32 smithi097 ceph-mon[67407]: Reconfiguring daemon mon.c on smithi097 2024-04-12T22:03:32.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:32 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:32.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:32 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:32.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:32 smithi097 ceph-mon[64853]: Reconfiguring mon.c (monmap changed)... 2024-04-12T22:03:32.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:32 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T22:03:32.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:32 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T22:03:32.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:32 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:32.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:32 smithi097 ceph-mon[64853]: Reconfiguring daemon mon.c on smithi097 2024-04-12T22:03:33.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:33 smithi114 ceph-mon[58780]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-12T22:03:33.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:33 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:33.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:33 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:33.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:33 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-12T22:03:33.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:33 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:33.853 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:33 smithi097 ceph-mon[64853]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-12T22:03:33.853 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:33 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:33.853 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:33 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:33.853 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:33 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-12T22:03:33.853 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:33 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:33.853 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:33 smithi097 ceph-mon[67407]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-12T22:03:33.853 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:33 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:33.854 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:33 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:33.854 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:33 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-12T22:03:33.854 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:33 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:34.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:34 smithi097 ceph-mon[64853]: Reconfiguring osd.0 (monmap changed)... 2024-04-12T22:03:34.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:34 smithi097 ceph-mon[64853]: Reconfiguring daemon osd.0 on smithi097 2024-04-12T22:03:34.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:34 smithi097 ceph-mon[67407]: Reconfiguring osd.0 (monmap changed)... 2024-04-12T22:03:34.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:34 smithi097 ceph-mon[67407]: Reconfiguring daemon osd.0 on smithi097 2024-04-12T22:03:34.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:34 smithi114 ceph-mon[58780]: Reconfiguring osd.0 (monmap changed)... 2024-04-12T22:03:34.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:34 smithi114 ceph-mon[58780]: Reconfiguring daemon osd.0 on smithi097 2024-04-12T22:03:36.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:35 smithi097 ceph-mon[64853]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-12T22:03:36.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:35 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:36.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:35 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:36.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:35 smithi097 ceph-mon[64853]: Reconfiguring osd.1 (monmap changed)... 2024-04-12T22:03:36.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:35 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-12T22:03:36.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:35 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:36.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:35 smithi097 ceph-mon[64853]: Reconfiguring daemon osd.1 on smithi097 2024-04-12T22:03:36.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:35 smithi097 ceph-mon[67407]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-12T22:03:36.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:35 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:36.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:35 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:36.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:35 smithi097 ceph-mon[67407]: Reconfiguring osd.1 (monmap changed)... 2024-04-12T22:03:36.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:35 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-12T22:03:36.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:35 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:36.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:35 smithi097 ceph-mon[67407]: Reconfiguring daemon osd.1 on smithi097 2024-04-12T22:03:36.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:35 smithi114 ceph-mon[58780]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-12T22:03:36.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:35 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:36.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:35 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:36.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:35 smithi114 ceph-mon[58780]: Reconfiguring osd.1 (monmap changed)... 2024-04-12T22:03:36.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:35 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-12T22:03:36.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:35 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:36.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:35 smithi114 ceph-mon[58780]: Reconfiguring daemon osd.1 on smithi097 2024-04-12T22:03:37.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:37 smithi097 ceph-mon[64853]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-12T22:03:37.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:37 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:37.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:37 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:37.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:37 smithi097 ceph-mon[64853]: Reconfiguring osd.2 (monmap changed)... 2024-04-12T22:03:37.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:37 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-12T22:03:37.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:37 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:37.500 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:37 smithi097 ceph-mon[64853]: Reconfiguring daemon osd.2 on smithi097 2024-04-12T22:03:37.501 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:37 smithi097 ceph-mon[67407]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-12T22:03:37.501 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:37 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:37.501 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:37 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:37.501 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:37 smithi097 ceph-mon[67407]: Reconfiguring osd.2 (monmap changed)... 2024-04-12T22:03:37.501 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:37 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-12T22:03:37.501 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:37 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:37.501 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:37 smithi097 ceph-mon[67407]: Reconfiguring daemon osd.2 on smithi097 2024-04-12T22:03:37.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:37 smithi114 ceph-mon[58780]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-12T22:03:37.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:37 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:37.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:37 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:37.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:37 smithi114 ceph-mon[58780]: Reconfiguring osd.2 (monmap changed)... 2024-04-12T22:03:37.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:37 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-12T22:03:37.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:37 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:37.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:37 smithi114 ceph-mon[58780]: Reconfiguring daemon osd.2 on smithi097 2024-04-12T22:03:38.993 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:03:38 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:03:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"smithi097:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"172.21.15.114:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-12T22:03:39.410 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:03:39 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:03:39] "GET /metrics HTTP/1.1" 200 38275 "" "Prometheus/2.33.4" 2024-04-12T22:03:39.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:39 smithi097 ceph-mon[67407]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-12T22:03:39.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:39 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:39.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:39 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:39.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:39 smithi097 ceph-mon[67407]: Reconfiguring osd.3 (monmap changed)... 2024-04-12T22:03:39.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:39 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-12T22:03:39.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:39 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:39.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:39 smithi097 ceph-mon[67407]: Reconfiguring daemon osd.3 on smithi097 2024-04-12T22:03:39.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:39 smithi097 ceph-mon[64853]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-12T22:03:39.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:39 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:39.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:39 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:39.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:39 smithi097 ceph-mon[64853]: Reconfiguring osd.3 (monmap changed)... 2024-04-12T22:03:39.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:39 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-12T22:03:39.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:39 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:39.412 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:39 smithi097 ceph-mon[64853]: Reconfiguring daemon osd.3 on smithi097 2024-04-12T22:03:39.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:39 smithi114 ceph-mon[58780]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-12T22:03:39.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:39 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:39.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:39 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:39.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:39 smithi114 ceph-mon[58780]: Reconfiguring osd.3 (monmap changed)... 2024-04-12T22:03:39.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:39 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-12T22:03:39.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:39 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:39.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:39 smithi114 ceph-mon[58780]: Reconfiguring daemon osd.3 on smithi097 2024-04-12T22:03:40.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:40 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:03:40.422 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:40 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:40.423 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:40 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:40.423 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:40 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi097.sggtkh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-12T22:03:40.423 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:40 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:40.423 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:40 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:03:40.423 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:40 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:40.423 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:40 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:40.423 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:40 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi097.sggtkh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-12T22:03:40.423 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:40 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:40.493 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:40 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@grafana.a.service: Scheduled restart job, restart counter is at 11. 2024-04-12T22:03:40.493 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:40 smithi114 systemd[1]: Stopped Ceph grafana.a for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:03:40.493 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:40 smithi114 systemd[1]: Starting Ceph grafana.a for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:03:40.493 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:40 smithi114 podman[60054]: 2024-04-12 22:03:40.382862479 +0000 UTC m=+0.017234991 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-04-12T22:03:40.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:40 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:03:40.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:40 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:40.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:40 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:40.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:40 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi097.sggtkh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-12T22:03:40.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:40 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:40.785 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:40 smithi114 podman[60054]: 2024-04-12 22:03:40.519057909 +0000 UTC m=+0.153430420 container create c1c1c9c837dab4c5170c630fafd21786159e5a76e189f2b5e2b7e814c49904ff (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, maintainer=Paul Cuzner , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=Red Hat Universal Base Image 8, build-date=2022-03-28T10:36:18.413762, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.expose-services=, vendor=Red Hat, Inc., io.openshift.tags=base rhel8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.24.2, vcs-type=git, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, com.redhat.component=ubi8-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, architecture=x86_64, name=ubi8, release=236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public, description=Ceph Grafana Container) 2024-04-12T22:03:40.785 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:40 smithi114 podman[60054]: 2024-04-12 22:03:40.754443599 +0000 UTC m=+0.388816109 container init c1c1c9c837dab4c5170c630fafd21786159e5a76e189f2b5e2b7e814c49904ff (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=ubi8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, version=8.5, maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, io.buildah.version=1.24.2, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, com.redhat.component=ubi8-container, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, build-date=2022-03-28T10:36:18.413762, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.expose-services=, release=236.1648460182, distribution-scope=public) 2024-04-12T22:03:40.785 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:40 smithi114 podman[60054]: 2024-04-12 22:03:40.757127327 +0000 UTC m=+0.391499846 container start c1c1c9c837dab4c5170c630fafd21786159e5a76e189f2b5e2b7e814c49904ff (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.expose-services=, version=8.5, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-type=git, build-date=2022-03-28T10:36:18.413762, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.display-name=Red Hat Universal Base Image 8, description=Ceph Grafana Container, io.openshift.tags=base rhel8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., io.buildah.version=1.24.2, maintainer=Paul Cuzner , release=236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.component=ubi8-container, name=ubi8) 2024-04-12T22:03:41.182 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:40 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[60069]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-12T22:03:41.182 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:40 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[60069]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-12T22:03:41.182 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:40 smithi114 bash[60054]: c1c1c9c837dab4c5170c630fafd21786159e5a76e189f2b5e2b7e814c49904ff 2024-04-12T22:03:41.182 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:40 smithi114 systemd[1]: Started Ceph grafana.a for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:03:41.182 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:40 smithi114 podman[60085]: 2024-04-12 22:03:40.889406174 +0000 UTC m=+0.097770378 container died c1c1c9c837dab4c5170c630fafd21786159e5a76e189f2b5e2b7e814c49904ff (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, name=ubi8, io.openshift.expose-services=, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, version=8.5, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-03-28T10:36:18.413762, maintainer=Paul Cuzner , vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vendor=Red Hat, Inc., io.buildah.version=1.24.2, architecture=x86_64, com.redhat.component=ubi8-container, io.openshift.tags=base rhel8, io.k8s.display-name=Red Hat Universal Base Image 8, description=Ceph Grafana Container, distribution-scope=public, release=236.1648460182) 2024-04-12T22:03:41.743 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:41 smithi114 podman[60085]: 2024-04-12 22:03:41.450305522 +0000 UTC m=+0.658669736 container remove c1c1c9c837dab4c5170c630fafd21786159e5a76e189f2b5e2b7e814c49904ff (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, distribution-scope=public, com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, build-date=2022-03-28T10:36:18.413762, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, architecture=x86_64, io.buildah.version=1.24.2, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, vendor=Red Hat, Inc., maintainer=Paul Cuzner , io.openshift.expose-services=, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container, io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8, version=8.5, summary=Grafana Container configured for Ceph mgr/dashboard integration, release=236.1648460182) 2024-04-12T22:03:41.743 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:41 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-04-12T22:03:41.744 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:41 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-04-12T22:03:42.064 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:41 smithi097 ceph-mon[64853]: Reconfiguring rgw.foo.smithi097.sggtkh (monmap changed)... 2024-04-12T22:03:42.064 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:41 smithi097 ceph-mon[64853]: Reconfiguring daemon rgw.foo.smithi097.sggtkh on smithi097 2024-04-12T22:03:42.064 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:41 smithi097 ceph-mon[64853]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:03:42.064 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:41 smithi097 ceph-mon[64853]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:03:42.064 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:41 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:42.064 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:41 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:42.064 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:41 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi097.exxkgl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-12T22:03:42.064 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:41 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:42.064 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:41 smithi097 ceph-mon[67407]: Reconfiguring rgw.foo.smithi097.sggtkh (monmap changed)... 2024-04-12T22:03:42.064 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:41 smithi097 ceph-mon[67407]: Reconfiguring daemon rgw.foo.smithi097.sggtkh on smithi097 2024-04-12T22:03:42.065 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:41 smithi097 ceph-mon[67407]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:03:42.065 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:41 smithi097 ceph-mon[67407]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:03:42.065 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:41 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:42.065 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:41 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:42.065 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:41 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi097.exxkgl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-12T22:03:42.065 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:41 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:42.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:41 smithi114 ceph-mon[58780]: Reconfiguring rgw.foo.smithi097.sggtkh (monmap changed)... 2024-04-12T22:03:42.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:41 smithi114 ceph-mon[58780]: Reconfiguring daemon rgw.foo.smithi097.sggtkh on smithi097 2024-04-12T22:03:42.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:41 smithi114 ceph-mon[58780]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:03:42.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:41 smithi114 ceph-mon[58780]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:03:42.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:41 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:42.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:41 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:42.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:41 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi097.exxkgl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-12T22:03:42.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:41 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:42.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:42 smithi097 ceph-mon[64853]: Reconfiguring rgw.smpl.smithi097.exxkgl (monmap changed)... 2024-04-12T22:03:42.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:42 smithi097 ceph-mon[64853]: Reconfiguring daemon rgw.smpl.smithi097.exxkgl on smithi097 2024-04-12T22:03:42.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:42 smithi097 ceph-mon[67407]: Reconfiguring rgw.smpl.smithi097.exxkgl (monmap changed)... 2024-04-12T22:03:42.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:42 smithi097 ceph-mon[67407]: Reconfiguring daemon rgw.smpl.smithi097.exxkgl on smithi097 2024-04-12T22:03:43.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:42 smithi114 ceph-mon[58780]: Reconfiguring rgw.smpl.smithi097.exxkgl (monmap changed)... 2024-04-12T22:03:43.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:42 smithi114 ceph-mon[58780]: Reconfiguring daemon rgw.smpl.smithi097.exxkgl on smithi097 2024-04-12T22:03:44.095 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:43 smithi114 ceph-mon[58780]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:03:44.095 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:43 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:44.095 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:43 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:44.096 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:43 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T22:03:44.096 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:43 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T22:03:44.096 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:43 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:44.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:43 smithi097 ceph-mon[64853]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:03:44.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:43 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:44.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:43 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:44.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:43 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T22:03:44.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:43 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T22:03:44.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:43 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:44.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:43 smithi097 ceph-mon[67407]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:03:44.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:43 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:44.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:43 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:44.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:43 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T22:03:44.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:43 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T22:03:44.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:43 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:44.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:44 smithi114 ceph-mon[58780]: Reconfiguring mon.b (monmap changed)... 2024-04-12T22:03:44.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:44 smithi114 ceph-mon[58780]: Reconfiguring daemon mon.b on smithi114 2024-04-12T22:03:44.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:44 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:44.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:44 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:44.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:44 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T22:03:44.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:44 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T22:03:44.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:44 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:45.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:44 smithi097 ceph-mon[64853]: Reconfiguring mon.b (monmap changed)... 2024-04-12T22:03:45.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:44 smithi097 ceph-mon[64853]: Reconfiguring daemon mon.b on smithi114 2024-04-12T22:03:45.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:44 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:45.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:44 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:45.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:44 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T22:03:45.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:44 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T22:03:45.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:44 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:45.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:44 smithi097 ceph-mon[67407]: Reconfiguring mon.b (monmap changed)... 2024-04-12T22:03:45.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:44 smithi097 ceph-mon[67407]: Reconfiguring daemon mon.b on smithi114 2024-04-12T22:03:45.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:44 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:45.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:44 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:45.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:44 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T22:03:45.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:44 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T22:03:45.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:44 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:45.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:45 smithi114 ceph-mon[58780]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:03:45.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:45 smithi114 ceph-mon[58780]: Reconfiguring mgr.x (monmap changed)... 2024-04-12T22:03:45.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:45 smithi114 ceph-mon[58780]: Reconfiguring daemon mgr.x on smithi114 2024-04-12T22:03:46.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:45 smithi097 ceph-mon[67407]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:03:46.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:45 smithi097 ceph-mon[67407]: Reconfiguring mgr.x (monmap changed)... 2024-04-12T22:03:46.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:45 smithi097 ceph-mon[67407]: Reconfiguring daemon mgr.x on smithi114 2024-04-12T22:03:46.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:45 smithi097 ceph-mon[64853]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:03:46.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:45 smithi097 ceph-mon[64853]: Reconfiguring mgr.x (monmap changed)... 2024-04-12T22:03:46.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:45 smithi097 ceph-mon[64853]: Reconfiguring daemon mgr.x on smithi114 2024-04-12T22:03:47.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:47 smithi114 ceph-mon[58780]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:03:47.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:47 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:47.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:47 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:47.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:47 smithi114 ceph-mon[58780]: Reconfiguring osd.4 (monmap changed)... 2024-04-12T22:03:47.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:47 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-12T22:03:47.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:47 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:47.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:47 smithi114 ceph-mon[58780]: Reconfiguring daemon osd.4 on smithi114 2024-04-12T22:03:47.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:47 smithi097 ceph-mon[64853]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:03:47.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:47 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:47.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:47 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:47.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:47 smithi097 ceph-mon[64853]: Reconfiguring osd.4 (monmap changed)... 2024-04-12T22:03:47.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:47 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-12T22:03:47.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:47 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:47.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:47 smithi097 ceph-mon[64853]: Reconfiguring daemon osd.4 on smithi114 2024-04-12T22:03:47.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:47 smithi097 ceph-mon[67407]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:03:47.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:47 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:47.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:47 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:47.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:47 smithi097 ceph-mon[67407]: Reconfiguring osd.4 (monmap changed)... 2024-04-12T22:03:47.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:47 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-12T22:03:47.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:47 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:47.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:47 smithi097 ceph-mon[67407]: Reconfiguring daemon osd.4 on smithi114 2024-04-12T22:03:48.831 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:03:48 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:03:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"smithi097:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"172.21.15.114:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-12T22:03:49.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:48 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:49.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:48 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:49.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:48 smithi114 ceph-mon[58780]: Reconfiguring osd.5 (monmap changed)... 2024-04-12T22:03:49.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:48 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-12T22:03:49.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:48 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:49.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:48 smithi114 ceph-mon[58780]: Reconfiguring daemon osd.5 on smithi114 2024-04-12T22:03:49.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:48 smithi114 ceph-mon[58780]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:03:49.410 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:03:49 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:03:49] "GET /metrics HTTP/1.1" 200 38273 "" "Prometheus/2.33.4" 2024-04-12T22:03:49.410 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:48 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:49.410 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:48 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:49.410 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:48 smithi097 ceph-mon[67407]: Reconfiguring osd.5 (monmap changed)... 2024-04-12T22:03:49.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:48 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-12T22:03:49.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:48 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:49.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:48 smithi097 ceph-mon[67407]: Reconfiguring daemon osd.5 on smithi114 2024-04-12T22:03:49.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:48 smithi097 ceph-mon[67407]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:03:49.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:48 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:49.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:48 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:49.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:48 smithi097 ceph-mon[64853]: Reconfiguring osd.5 (monmap changed)... 2024-04-12T22:03:49.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:48 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-12T22:03:49.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:48 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:49.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:48 smithi097 ceph-mon[64853]: Reconfiguring daemon osd.5 on smithi114 2024-04-12T22:03:49.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:48 smithi097 ceph-mon[64853]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:03:50.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:50 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:50.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:50 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:50.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:50 smithi114 ceph-mon[58780]: Reconfiguring osd.6 (monmap changed)... 2024-04-12T22:03:50.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:50 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-12T22:03:50.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:50 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:50.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:50 smithi114 ceph-mon[58780]: Reconfiguring daemon osd.6 on smithi114 2024-04-12T22:03:51.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:50 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:51.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:50 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:51.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:50 smithi097 ceph-mon[67407]: Reconfiguring osd.6 (monmap changed)... 2024-04-12T22:03:51.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:50 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-12T22:03:51.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:50 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:51.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:50 smithi097 ceph-mon[67407]: Reconfiguring daemon osd.6 on smithi114 2024-04-12T22:03:51.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:50 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:51.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:50 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:51.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:50 smithi097 ceph-mon[64853]: Reconfiguring osd.6 (monmap changed)... 2024-04-12T22:03:51.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:50 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-12T22:03:51.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:50 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:51.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:50 smithi097 ceph-mon[64853]: Reconfiguring daemon osd.6 on smithi114 2024-04-12T22:03:51.848 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:51 smithi114 ceph-mon[58780]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:03:51.848 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:51 smithi114 ceph-mon[58780]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:03:51.848 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:51 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:51.848 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:51 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:51.848 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:51 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-12T22:03:51.848 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:51 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:51.848 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:51 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@grafana.a.service: Scheduled restart job, restart counter is at 12. 2024-04-12T22:03:51.848 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:51 smithi114 systemd[1]: Stopped Ceph grafana.a for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:03:51.849 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:51 smithi114 systemd[1]: Starting Ceph grafana.a for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:03:51.849 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:51 smithi114 podman[60874]: 2024-04-12 22:03:51.627649171 +0000 UTC m=+0.019196317 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-04-12T22:03:51.849 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:51 smithi114 podman[60874]: 2024-04-12 22:03:51.725483277 +0000 UTC m=+0.117030400 container create 3f2c5beebaf9428cab719466df560ecd8ef8a5877a08faf56533fe2c9e8f45b4 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, release=236.1648460182, version=8.5, description=Ceph Grafana Container, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.component=ubi8-container, build-date=2022-03-28T10:36:18.413762, summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.display-name=Red Hat Universal Base Image 8, distribution-scope=public, io.openshift.expose-services=, io.buildah.version=1.24.2, name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner ) 2024-04-12T22:03:52.127 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:51 smithi097 ceph-mon[64853]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:03:52.127 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:51 smithi097 ceph-mon[64853]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:03:52.127 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:51 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:52.127 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:51 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:52.127 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:51 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-12T22:03:52.127 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:51 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:52.128 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:51 smithi097 ceph-mon[67407]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:03:52.128 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:51 smithi097 ceph-mon[67407]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:03:52.128 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:51 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:52.128 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:51 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:52.128 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:51 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-12T22:03:52.128 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:51 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:52.244 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:51 smithi114 podman[60874]: 2024-04-12 22:03:51.986336428 +0000 UTC m=+0.377883552 container init 3f2c5beebaf9428cab719466df560ecd8ef8a5877a08faf56533fe2c9e8f45b4 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, version=8.5, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, release=236.1648460182, io.buildah.version=1.24.2, description=Ceph Grafana Container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, name=ubi8, io.openshift.expose-services=, io.openshift.tags=base rhel8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.component=ubi8-container, io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , distribution-scope=public, vcs-type=git, build-date=2022-03-28T10:36:18.413762, summary=Grafana Container configured for Ceph mgr/dashboard integration) 2024-04-12T22:03:52.244 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:51 smithi114 podman[60874]: 2024-04-12 22:03:51.989691928 +0000 UTC m=+0.381239051 container start 3f2c5beebaf9428cab719466df560ecd8ef8a5877a08faf56533fe2c9e8f45b4 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Paul Cuzner , io.openshift.expose-services=, name=ubi8, io.openshift.tags=base rhel8, version=8.5, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.component=ubi8-container, distribution-scope=public, vcs-type=git, summary=Grafana Container configured for Ceph mgr/dashboard integration, build-date=2022-03-28T10:36:18.413762, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, description=Ceph Grafana Container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vendor=Red Hat, Inc., io.buildah.version=1.24.2, architecture=x86_64, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, release=236.1648460182, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-12T22:03:52.244 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[60901]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-12T22:03:52.244 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[60901]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-12T22:03:52.244 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:52 smithi114 bash[60874]: 3f2c5beebaf9428cab719466df560ecd8ef8a5877a08faf56533fe2c9e8f45b4 2024-04-12T22:03:52.244 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:52 smithi114 systemd[1]: Started Ceph grafana.a for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:03:52.244 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:52 smithi114 podman[60935]: 2024-04-12 22:03:52.094877064 +0000 UTC m=+0.071846608 container died 3f2c5beebaf9428cab719466df560ecd8ef8a5877a08faf56533fe2c9e8f45b4 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, io.k8s.display-name=Red Hat Universal Base Image 8, io.buildah.version=1.24.2, vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.component=ubi8-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, summary=Grafana Container configured for Ceph mgr/dashboard integration, name=ubi8, io.openshift.tags=base rhel8, maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, distribution-scope=public, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container, version=8.5, build-date=2022-03-28T10:36:18.413762, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, release=236.1648460182) 2024-04-12T22:03:52.942 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:52 smithi114 podman[60935]: 2024-04-12 22:03:52.664779618 +0000 UTC m=+0.641749170 container remove 3f2c5beebaf9428cab719466df560ecd8ef8a5877a08faf56533fe2c9e8f45b4 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, io.openshift.tags=base rhel8, vcs-type=git, description=Ceph Grafana Container, name=ubi8, io.buildah.version=1.24.2, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.openshift.expose-services=, maintainer=Paul Cuzner , com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.component=ubi8-container, io.k8s.display-name=Red Hat Universal Base Image 8, build-date=2022-03-28T10:36:18.413762, vendor=Red Hat, Inc., version=8.5, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, summary=Grafana Container configured for Ceph mgr/dashboard integration, release=236.1648460182) 2024-04-12T22:03:52.942 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:52 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-04-12T22:03:52.942 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:03:52 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-04-12T22:03:52.942 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:52 smithi114 ceph-mon[58780]: Reconfiguring osd.7 (monmap changed)... 2024-04-12T22:03:52.943 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:52 smithi114 ceph-mon[58780]: Reconfiguring daemon osd.7 on smithi114 2024-04-12T22:03:53.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:52 smithi097 ceph-mon[64853]: Reconfiguring osd.7 (monmap changed)... 2024-04-12T22:03:53.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:52 smithi097 ceph-mon[64853]: Reconfiguring daemon osd.7 on smithi114 2024-04-12T22:03:53.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:52 smithi097 ceph-mon[67407]: Reconfiguring osd.7 (monmap changed)... 2024-04-12T22:03:53.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:52 smithi097 ceph-mon[67407]: Reconfiguring daemon osd.7 on smithi114 2024-04-12T22:03:53.788 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:53 smithi114 ceph-mon[58780]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:03:54.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:53 smithi097 ceph-mon[64853]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:03:54.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:53 smithi097 ceph-mon[67407]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:03:55.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:54 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:55.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:54 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:55.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:54 smithi097 ceph-mon[64853]: Reconfiguring rgw.foo.smithi114.ulfbyo (monmap changed)... 2024-04-12T22:03:55.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:54 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi114.ulfbyo", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-12T22:03:55.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:54 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:55.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:54 smithi097 ceph-mon[64853]: Reconfiguring daemon rgw.foo.smithi114.ulfbyo on smithi114 2024-04-12T22:03:55.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:54 smithi097 ceph-mon[64853]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:03:55.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:54 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:03:55.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:54 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:55.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:54 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:55.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:54 smithi097 ceph-mon[67407]: Reconfiguring rgw.foo.smithi114.ulfbyo (monmap changed)... 2024-04-12T22:03:55.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:54 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi114.ulfbyo", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-12T22:03:55.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:54 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:55.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:54 smithi097 ceph-mon[67407]: Reconfiguring daemon rgw.foo.smithi114.ulfbyo on smithi114 2024-04-12T22:03:55.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:54 smithi097 ceph-mon[67407]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:03:55.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:54 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:03:55.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:54 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:55.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:54 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:55.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:54 smithi114 ceph-mon[58780]: Reconfiguring rgw.foo.smithi114.ulfbyo (monmap changed)... 2024-04-12T22:03:55.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:54 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi114.ulfbyo", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-12T22:03:55.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:54 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:55.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:54 smithi114 ceph-mon[58780]: Reconfiguring daemon rgw.foo.smithi114.ulfbyo on smithi114 2024-04-12T22:03:55.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:54 smithi114 ceph-mon[58780]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:03:55.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:54 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:03:56.736 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:56 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:56.736 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:56 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:56.736 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:56 smithi114 ceph-mon[58780]: Reconfiguring rgw.smpl.smithi114.lrgvph (monmap changed)... 2024-04-12T22:03:56.736 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:56 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi114.lrgvph", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-12T22:03:56.736 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:56 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:56.736 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:56 smithi114 ceph-mon[58780]: Reconfiguring daemon rgw.smpl.smithi114.lrgvph on smithi114 2024-04-12T22:03:56.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:56 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:56.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:56 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:56.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:56 smithi097 ceph-mon[64853]: Reconfiguring rgw.smpl.smithi114.lrgvph (monmap changed)... 2024-04-12T22:03:56.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:56 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi114.lrgvph", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-12T22:03:56.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:56 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:56.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:56 smithi097 ceph-mon[64853]: Reconfiguring daemon rgw.smpl.smithi114.lrgvph on smithi114 2024-04-12T22:03:56.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:56 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:56.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:56 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:56.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:56 smithi097 ceph-mon[67407]: Reconfiguring rgw.smpl.smithi114.lrgvph (monmap changed)... 2024-04-12T22:03:56.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:56 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi114.lrgvph", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-12T22:03:56.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:56 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:56.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:56 smithi097 ceph-mon[67407]: Reconfiguring daemon rgw.smpl.smithi114.lrgvph on smithi114 2024-04-12T22:03:57.085 INFO:teuthology.orchestra.run.smithi097.stdout:true 2024-04-12T22:03:57.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:57 smithi114 ceph-mon[58780]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:03:57.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:57 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:57.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:57 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:57.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:57 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:03:57.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:57 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:03:57.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:57 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:03:57.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:57 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:57.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:57 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-12T22:03:57.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:57 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-04-12T22:03:57.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:57 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-12T22:03:57.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:57 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-04-12T22:03:57.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:57 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-12T22:03:57.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:57 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-04-12T22:03:57.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:57 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:03:57.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:57 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:57.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:57 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-12T22:03:57.774 INFO:teuthology.orchestra.run.smithi097.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T22:03:57.775 INFO:teuthology.orchestra.run.smithi097.stdout:alertmanager.a smithi097 *:9093,9094 running (2m) 55s ago 7m 17.8M - 0.23.0 ba2b418f427c 7b4c491fe0a0 2024-04-12T22:03:57.775 INFO:teuthology.orchestra.run.smithi097.stdout:grafana.a smithi114 *:3000 unknown 35s ago 7m - - 2024-04-12T22:03:57.775 INFO:teuthology.orchestra.run.smithi097.stdout:iscsi.foo.smithi097.xndkxy smithi097 running (114s) 55s ago 6m 47.4M - 3.5 e1d6a67b021e 0436463f28cd 2024-04-12T22:03:57.775 INFO:teuthology.orchestra.run.smithi097.stdout:mgr.x smithi114 *:8443,9283,8765 running (103s) 35s ago 12m 450M - 19.0.0-2053-gb277feb2 668895944f0e b8f8df3bf928 2024-04-12T22:03:57.775 INFO:teuthology.orchestra.run.smithi097.stdout:mgr.y smithi097 *:8443,9283,8765 running (2m) 55s ago 14m 494M - 19.0.0-2053-gb277feb2 668895944f0e 488170dac11f 2024-04-12T22:03:57.775 INFO:teuthology.orchestra.run.smithi097.stdout:mon.a smithi097 running (88s) 55s ago 14m 41.1M 2048M 19.0.0-2053-gb277feb2 668895944f0e 3cfa16d3a85c 2024-04-12T22:03:57.775 INFO:teuthology.orchestra.run.smithi097.stdout:mon.b smithi114 running (45s) 35s ago 12m 18.4M 2048M 19.0.0-2053-gb277feb2 668895944f0e 71a2eff159a3 2024-04-12T22:03:57.775 INFO:teuthology.orchestra.run.smithi097.stdout:mon.c smithi097 running (62s) 55s ago 12m 18.0M 2048M 19.0.0-2053-gb277feb2 668895944f0e 69507ba88603 2024-04-12T22:03:57.775 INFO:teuthology.orchestra.run.smithi097.stdout:node-exporter.a smithi097 *:9100 running (3m) 55s ago 8m 19.2M - 1.3.1 1dbe0e931976 5edff9053b37 2024-04-12T22:03:57.775 INFO:teuthology.orchestra.run.smithi097.stdout:node-exporter.b smithi114 *:9100 running (2m) 35s ago 8m 17.0M - 1.3.1 1dbe0e931976 904afc45dc2f 2024-04-12T22:03:57.775 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0 smithi097 running (11m) 55s ago 11m 55.0M 2736M 17.2.0 e1d6a67b021e abac1d8884fa 2024-04-12T22:03:57.775 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1 smithi097 running (11m) 55s ago 11m 54.3M 2736M 17.2.0 e1d6a67b021e 39192562d598 2024-04-12T22:03:57.775 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2 smithi097 running (11m) 55s ago 11m 51.3M 2736M 17.2.0 e1d6a67b021e 89daf8e339c3 2024-04-12T22:03:57.776 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3 smithi097 running (10m) 55s ago 10m 51.8M 2736M 17.2.0 e1d6a67b021e 4e9a0b3862d9 2024-04-12T22:03:57.776 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4 smithi114 running (10m) 35s ago 10m 53.6M 2992M 17.2.0 e1d6a67b021e fb42c0e8e575 2024-04-12T22:03:57.776 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5 smithi114 running (10m) 35s ago 10m 50.2M 2992M 17.2.0 e1d6a67b021e 3acd4a3b61e5 2024-04-12T22:03:57.776 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6 smithi114 running (9m) 35s ago 9m 49.6M 2992M 17.2.0 e1d6a67b021e 5969baf67f41 2024-04-12T22:03:57.776 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7 smithi114 running (9m) 35s ago 9m 51.8M 2992M 17.2.0 e1d6a67b021e f68340fecf2c 2024-04-12T22:03:57.776 INFO:teuthology.orchestra.run.smithi097.stdout:prometheus.a smithi114 *:9095 running (109s) 35s ago 8m 65.1M - 2.33.4 514e6a882f6e 0ad5437ed63f 2024-04-12T22:03:57.776 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.foo.smithi097.sggtkh smithi097 *:8000 running (7m) 55s ago 7m 72.2M - 17.2.0 e1d6a67b021e a39f9b2d8509 2024-04-12T22:03:57.776 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.foo.smithi114.ulfbyo smithi114 *:8000 running (6m) 35s ago 6m 72.8M - 17.2.0 e1d6a67b021e df79a48bdd59 2024-04-12T22:03:57.776 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.smpl.smithi097.exxkgl smithi097 *:80 running (6m) 55s ago 6m 72.2M - 17.2.0 e1d6a67b021e a036c58a0099 2024-04-12T22:03:57.776 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.smpl.smithi114.lrgvph smithi114 *:80 running (6m) 35s ago 6m 74.4M - 17.2.0 e1d6a67b021e 031a5fc24237 2024-04-12T22:03:57.776 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:57 smithi097 ceph-mon[67407]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:03:57.777 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:57 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:57.777 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:57 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:57.777 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:57 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:03:57.777 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:57 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:03:57.777 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:57 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:03:57.777 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:57 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:57.777 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:57 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-12T22:03:57.777 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:57 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-04-12T22:03:57.777 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:57 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-12T22:03:57.777 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:57 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-04-12T22:03:57.777 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:57 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-12T22:03:57.777 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:57 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-04-12T22:03:57.777 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:57 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:03:57.777 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:57 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:57.777 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:57 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-12T22:03:57.778 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:57 smithi097 ceph-mon[64853]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:03:57.778 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:57 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:57.778 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:57 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:57.778 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:57 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:03:57.778 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:57 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:03:57.778 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:57 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:03:57.778 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:57 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:57.778 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:57 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-12T22:03:57.778 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:57 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-04-12T22:03:57.778 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:57 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-12T22:03:57.778 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:57 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-04-12T22:03:57.778 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:57 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-12T22:03:57.779 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:57 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-04-12T22:03:57.779 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:57 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:03:57.779 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:57 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:57.779 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:57 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-12T22:03:58.181 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-12T22:03:58.181 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": { 2024-04-12T22:03:58.181 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 3 2024-04-12T22:03:58.181 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:03:58.181 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": { 2024-04-12T22:03:58.181 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 2 2024-04-12T22:03:58.181 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:03:58.181 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": { 2024-04-12T22:03:58.181 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-12T22:03:58.181 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:03:58.181 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw": { 2024-04-12T22:03:58.181 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-12T22:03:58.181 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:03:58.182 INFO:teuthology.orchestra.run.smithi097.stdout: "overall": { 2024-04-12T22:03:58.182 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-04-12T22:03:58.182 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 5 2024-04-12T22:03:58.182 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-12T22:03:58.182 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-12T22:03:58.516 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-12T22:03:58.516 INFO:teuthology.orchestra.run.smithi097.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a", 2024-04-12T22:03:58.517 INFO:teuthology.orchestra.run.smithi097.stdout: "in_progress": true, 2024-04-12T22:03:58.517 INFO:teuthology.orchestra.run.smithi097.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T22:03:58.517 INFO:teuthology.orchestra.run.smithi097.stdout: "services_complete": [ 2024-04-12T22:03:58.517 INFO:teuthology.orchestra.run.smithi097.stdout: "mon", 2024-04-12T22:03:58.517 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr" 2024-04-12T22:03:58.517 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-04-12T22:03:58.517 INFO:teuthology.orchestra.run.smithi097.stdout: "progress": "5/23 daemons upgraded", 2024-04-12T22:03:58.517 INFO:teuthology.orchestra.run.smithi097.stdout: "message": "Currently upgrading osd daemons", 2024-04-12T22:03:58.517 INFO:teuthology.orchestra.run.smithi097.stdout: "is_paused": false 2024-04-12T22:03:58.517 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-12T22:03:58.743 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:03:58 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:03:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"smithi097:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"172.21.15.114:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-12T22:03:58.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:58 smithi114 ceph-mon[58780]: from='client.44137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:03:58.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:58 smithi114 ceph-mon[58780]: Upgrade: Setting container_image for all mon 2024-04-12T22:03:58.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:58 smithi114 ceph-mon[58780]: Upgrade: Setting container_image for all crash 2024-04-12T22:03:58.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:58 smithi114 ceph-mon[58780]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-12T22:03:58.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:58 smithi114 ceph-mon[58780]: Upgrade: osd.0 is safe to restart 2024-04-12T22:03:58.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:58 smithi114 ceph-mon[58780]: from='client.44143 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:03:58.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:58 smithi114 ceph-mon[58780]: from='client.44149 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:03:58.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:58 smithi114 ceph-mon[58780]: from='client.? 172.21.15.97:0/1215534090' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:03:58.882 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:58 smithi097 ceph-mon[64853]: from='client.44137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:03:58.882 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:58 smithi097 ceph-mon[64853]: Upgrade: Setting container_image for all mon 2024-04-12T22:03:58.882 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:58 smithi097 ceph-mon[64853]: Upgrade: Setting container_image for all crash 2024-04-12T22:03:58.882 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:58 smithi097 ceph-mon[64853]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-12T22:03:58.882 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:58 smithi097 ceph-mon[64853]: Upgrade: osd.0 is safe to restart 2024-04-12T22:03:58.883 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:58 smithi097 ceph-mon[64853]: from='client.44143 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:03:58.883 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:58 smithi097 ceph-mon[64853]: from='client.44149 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:03:58.883 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:58 smithi097 ceph-mon[64853]: from='client.? 172.21.15.97:0/1215534090' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:03:58.883 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:58 smithi097 ceph-mon[67407]: from='client.44137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:03:58.883 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:58 smithi097 ceph-mon[67407]: Upgrade: Setting container_image for all mon 2024-04-12T22:03:58.883 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:58 smithi097 ceph-mon[67407]: Upgrade: Setting container_image for all crash 2024-04-12T22:03:58.883 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:58 smithi097 ceph-mon[67407]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-12T22:03:58.883 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:58 smithi097 ceph-mon[67407]: Upgrade: osd.0 is safe to restart 2024-04-12T22:03:58.883 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:58 smithi097 ceph-mon[67407]: from='client.44143 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:03:58.883 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:58 smithi097 ceph-mon[67407]: from='client.44149 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:03:58.883 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:58 smithi097 ceph-mon[67407]: from='client.? 172.21.15.97:0/1215534090' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:03:58.914 INFO:teuthology.orchestra.run.smithi097.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-12T22:03:58.915 INFO:teuthology.orchestra.run.smithi097.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-12T22:03:58.915 INFO:teuthology.orchestra.run.smithi097.stdout: daemon grafana.a on smithi114 is in unknown state 2024-04-12T22:03:59.486 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:03:59 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:03:59] "GET /metrics HTTP/1.1" 200 38267 "" "Prometheus/2.33.4" 2024-04-12T22:03:59.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:59 smithi114 ceph-mon[58780]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:03:59.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:59 smithi114 ceph-mon[58780]: from='client.44161 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:03:59.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:59 smithi114 ceph-mon[58780]: from='client.? 172.21.15.97:0/13513866' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T22:03:59.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:59 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:59.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:59 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-12T22:03:59.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:03:59 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:59.795 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:59 smithi097 ceph-mon[64853]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:03:59.795 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:59 smithi097 ceph-mon[64853]: from='client.44161 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:03:59.795 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:59 smithi097 ceph-mon[64853]: from='client.? 172.21.15.97:0/13513866' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T22:03:59.796 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:59 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:59.796 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:59 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-12T22:03:59.796 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:03:59 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:59.796 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:59 smithi097 ceph-mon[67407]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:03:59.796 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:59 smithi097 ceph-mon[67407]: from='client.44161 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:03:59.796 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:59 smithi097 ceph-mon[67407]: from='client.? 172.21.15.97:0/13513866' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T22:03:59.796 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:59 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:03:59.796 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:59 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-12T22:03:59.796 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:03:59 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:04:00.700 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:00 smithi097 ceph-mon[64853]: Upgrade: Updating osd.0 2024-04-12T22:04:00.701 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:00 smithi097 ceph-mon[64853]: Deploying daemon osd.0 on smithi097 2024-04-12T22:04:00.701 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:00 smithi097 ceph-mon[67407]: Upgrade: Updating osd.0 2024-04-12T22:04:00.701 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:00 smithi097 ceph-mon[67407]: Deploying daemon osd.0 on smithi097 2024-04-12T22:04:00.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:00 smithi114 ceph-mon[58780]: Upgrade: Updating osd.0 2024-04-12T22:04:00.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:00 smithi114 ceph-mon[58780]: Deploying daemon osd.0 on smithi097 2024-04-12T22:04:01.583 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:01 smithi097 ceph-mon[64853]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:04:01.583 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:01 smithi097 ceph-mon[64853]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:04:01.583 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:01 smithi097 ceph-mon[67407]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:04:01.583 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:01 smithi097 ceph-mon[67407]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:04:01.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:01 smithi114 ceph-mon[58780]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:04:01.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:01 smithi114 ceph-mon[58780]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:04:02.488 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:04:02 smithi097 systemd[1]: Stopping Ceph osd.0 for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:04:02.755 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:04:02 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@grafana.a.service: Scheduled restart job, restart counter is at 13. 2024-04-12T22:04:02.755 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:04:02 smithi114 systemd[1]: Stopped Ceph grafana.a for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:04:02.755 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:02 smithi114 ceph-mon[58780]: osd.0 marked itself down and dead 2024-04-12T22:04:02.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:02 smithi097 ceph-mon[64853]: osd.0 marked itself down and dead 2024-04-12T22:04:02.910 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:02 smithi097 ceph-mon[67407]: osd.0 marked itself down and dead 2024-04-12T22:04:02.911 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:04:02 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-0[32474]: 2024-04-12T22:04:02.486+0000 7fe71672c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-12T22:04:02.911 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:04:02 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-0[32474]: 2024-04-12T22:04:02.486+0000 7fe71672c700 -1 osd.0 106 *** Got signal Terminated *** 2024-04-12T22:04:02.911 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:04:02 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-0[32474]: 2024-04-12T22:04:02.486+0000 7fe71672c700 -1 osd.0 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-12T22:04:03.086 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:04:02 smithi114 systemd[1]: Starting Ceph grafana.a for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:04:03.086 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:04:02 smithi114 podman[61299]: 2024-04-12 22:04:02.881700918 +0000 UTC m=+0.017603488 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-04-12T22:04:03.086 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:04:02 smithi114 podman[61299]: 2024-04-12 22:04:02.994215985 +0000 UTC m=+0.130118537 container create 67192b51f599bc7ea8ad68f3b3c8d975bd5fb20101940c54df271cc4fbecf0ae (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, build-date=2022-03-28T10:36:18.413762, name=ubi8, io.buildah.version=1.24.2, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, maintainer=Paul Cuzner , com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=236.1648460182, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-type=git, distribution-scope=public, io.openshift.expose-services=, vendor=Red Hat, Inc., com.redhat.component=ubi8-container, description=Ceph Grafana Container, architecture=x86_64, version=8.5) 2024-04-12T22:04:03.239 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:04:02 smithi097 podman[70316]: 2024-04-12 22:04:02.956027928 +0000 UTC m=+0.567454124 container died abac1d8884fa2a5ad9463ecf84f2500778576b6ca384ce89288c22ae60c93403 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-0, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, GIT_BRANCH=HEAD, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., RELEASE=HEAD, io.buildah.version=1.19.8, GIT_CLEAN=True, version=8, io.openshift.expose-services=, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, io.k8s.display-name=CentOS Stream 8) 2024-04-12T22:04:03.493 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:04:03 smithi114 podman[61299]: 2024-04-12 22:04:03.213146334 +0000 UTC m=+0.349048886 container init 67192b51f599bc7ea8ad68f3b3c8d975bd5fb20101940c54df271cc4fbecf0ae (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, vcs-type=git, description=Ceph Grafana Container, name=ubi8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, release=236.1648460182, io.buildah.version=1.24.2, architecture=x86_64, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., io.openshift.tags=base rhel8, io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, version=8.5, maintainer=Paul Cuzner , com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, build-date=2022-03-28T10:36:18.413762) 2024-04-12T22:04:03.493 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:04:03 smithi114 podman[61299]: 2024-04-12 22:04:03.21620801 +0000 UTC m=+0.352110572 container start 67192b51f599bc7ea8ad68f3b3c8d975bd5fb20101940c54df271cc4fbecf0ae (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, build-date=2022-03-28T10:36:18.413762, vcs-type=git, io.openshift.tags=base rhel8, version=8.5, io.openshift.expose-services=, maintainer=Paul Cuzner , release=236.1648460182, com.redhat.component=ubi8-container, name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, distribution-scope=public, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, architecture=x86_64, io.buildah.version=1.24.2) 2024-04-12T22:04:03.494 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:04:03 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[61313]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-12T22:04:03.494 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:04:03 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a[61313]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-12T22:04:03.494 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:04:03 smithi114 bash[61299]: 67192b51f599bc7ea8ad68f3b3c8d975bd5fb20101940c54df271cc4fbecf0ae 2024-04-12T22:04:03.494 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:04:03 smithi114 systemd[1]: Started Ceph grafana.a for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:04:03.494 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:04:03 smithi114 podman[61329]: 2024-04-12 22:04:03.322047394 +0000 UTC m=+0.072441694 container died 67192b51f599bc7ea8ad68f3b3c8d975bd5fb20101940c54df271cc4fbecf0ae (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, vcs-type=git, description=Ceph Grafana Container, io.buildah.version=1.24.2, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-03-28T10:36:18.413762, io.openshift.tags=base rhel8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, maintainer=Paul Cuzner , name=ubi8, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=ubi8-container, architecture=x86_64, release=236.1648460182, distribution-scope=public, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Grafana Container configured for Ceph mgr/dashboard integration, version=8.5) 2024-04-12T22:04:03.497 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:04:03 smithi097 podman[70316]: 2024-04-12 22:04:03.337066758 +0000 UTC m=+0.948492947 container cleanup abac1d8884fa2a5ad9463ecf84f2500778576b6ca384ce89288c22ae60c93403 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-0, GIT_BRANCH=HEAD, GIT_CLEAN=True, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, version=8, io.openshift.expose-services=, release=754, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8) 2024-04-12T22:04:03.497 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:04:03 smithi097 bash[70316]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-0 2024-04-12T22:04:03.850 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:03 smithi114 ceph-mon[58780]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:04:03.850 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:03 smithi114 ceph-mon[58780]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-12T22:04:03.850 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:03 smithi114 ceph-mon[58780]: osdmap e107: 8 total, 7 up, 8 in 2024-04-12T22:04:03.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:03 smithi097 ceph-mon[64853]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:04:03.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:03 smithi097 ceph-mon[64853]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-12T22:04:03.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:03 smithi097 ceph-mon[64853]: osdmap e107: 8 total, 7 up, 8 in 2024-04-12T22:04:03.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:03 smithi097 ceph-mon[67407]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:04:03.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:03 smithi097 ceph-mon[67407]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-12T22:04:03.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:03 smithi097 ceph-mon[67407]: osdmap e107: 8 total, 7 up, 8 in 2024-04-12T22:04:03.911 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:04:03 smithi097 podman[70331]: 2024-04-12 22:04:03.638681952 +0000 UTC m=+0.675495925 container remove abac1d8884fa2a5ad9463ecf84f2500778576b6ca384ce89288c22ae60c93403 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-0, io.openshift.tags=base centos centos-stream, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, io.openshift.expose-services=, ceph=True, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, architecture=x86_64, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , distribution-scope=public, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-04-12T22:04:04.243 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:04:03 smithi114 podman[61329]: 2024-04-12 22:04:03.850251229 +0000 UTC m=+0.600645564 container remove 67192b51f599bc7ea8ad68f3b3c8d975bd5fb20101940c54df271cc4fbecf0ae (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-grafana-a, io.buildah.version=1.24.2, summary=Grafana Container configured for Ceph mgr/dashboard integration, version=8.5, vendor=Red Hat, Inc., name=ubi8, com.redhat.component=ubi8-container, release=236.1648460182, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Paul Cuzner , io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public, description=Ceph Grafana Container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-03-28T10:36:18.413762, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, io.openshift.expose-services=) 2024-04-12T22:04:04.243 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:04:03 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-04-12T22:04:04.243 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:04:03 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-04-12T22:04:04.271 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:04:03 smithi097 podman[70429]: 2024-04-12 22:04:03.859215279 +0000 UTC m=+0.017493623 image pull 668895944f0ec64b67a5b7564961d2756f2966e5d4b9338516300dbec9984951 quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a 2024-04-12T22:04:04.271 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:04:04 smithi097 podman[70429]: 2024-04-12 22:04:04.146419466 +0000 UTC m=+0.304697799 container create debe47552288593b873f513b3696f468ba51073e18e17b819fb92cf46ec347d1 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-b277feb, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-12T22:04:04.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:04 smithi097 ceph-mon[64853]: osdmap e108: 8 total, 7 up, 8 in 2024-04-12T22:04:04.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:04 smithi097 ceph-mon[67407]: osdmap e108: 8 total, 7 up, 8 in 2024-04-12T22:04:04.660 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:04:04 smithi097 podman[70429]: 2024-04-12 22:04:04.37558998 +0000 UTC m=+0.533868326 container init debe47552288593b873f513b3696f468ba51073e18e17b819fb92cf46ec347d1 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-b277feb, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-12T22:04:04.660 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:04:04 smithi097 podman[70429]: 2024-04-12 22:04:04.378783109 +0000 UTC m=+0.537061451 container start debe47552288593b873f513b3696f468ba51073e18e17b819fb92cf46ec347d1 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-b277feb) 2024-04-12T22:04:04.661 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:04:04 smithi097 podman[70429]: 2024-04-12 22:04:04.467258409 +0000 UTC m=+0.625536755 container attach debe47552288593b873f513b3696f468ba51073e18e17b819fb92cf46ec347d1 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-b277feb, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-12T22:04:04.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:04 smithi114 ceph-mon[58780]: osdmap e108: 8 total, 7 up, 8 in 2024-04-12T22:04:05.149 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:04:04 smithi097 podman[70429]: 2024-04-12 22:04:04.674571031 +0000 UTC m=+0.832849355 container died debe47552288593b873f513b3696f468ba51073e18e17b819fb92cf46ec347d1 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-b277feb, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-12T22:04:05.410 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:04:05 smithi097 podman[70455]: 2024-04-12 22:04:05.254458833 +0000 UTC m=+0.569973568 container remove debe47552288593b873f513b3696f468ba51073e18e17b819fb92cf46ec347d1 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-b277feb, org.label-schema.build-date=20240409) 2024-04-12T22:04:05.410 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:04:05 smithi097 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@osd.0.service: Deactivated successfully. 2024-04-12T22:04:05.410 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:04:05 smithi097 systemd[1]: Stopped Ceph osd.0 for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:04:05.410 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:04:05 smithi097 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@osd.0.service: Consumed 6.738s CPU time. 2024-04-12T22:04:05.739 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:05 smithi097 ceph-mon[64853]: pgmap v50: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-12T22:04:05.740 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:05 smithi097 ceph-mon[67407]: pgmap v50: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-12T22:04:05.740 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:04:05 smithi097 systemd[1]: Starting Ceph osd.0 for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:04:05.740 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:04:05 smithi097 podman[70558]: 2024-04-12 22:04:05.599446108 +0000 UTC m=+0.017777125 image pull 668895944f0ec64b67a5b7564961d2756f2966e5d4b9338516300dbec9984951 quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a 2024-04-12T22:04:05.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:05 smithi114 ceph-mon[58780]: pgmap v50: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-12T22:04:06.033 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:04:05 smithi097 podman[70558]: 2024-04-12 22:04:05.739017233 +0000 UTC m=+0.157348250 container create ed7a0de3252b5d661e49dfb254832ba1990db61bd9bb310a374b6d02e4a1a94e (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-0-activate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-b277feb) 2024-04-12T22:04:06.033 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:04:05 smithi097 podman[70558]: 2024-04-12 22:04:05.933124266 +0000 UTC m=+0.351455285 container init ed7a0de3252b5d661e49dfb254832ba1990db61bd9bb310a374b6d02e4a1a94e (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-0-activate, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-b277feb, maintainer=Guillaume Abrioux ) 2024-04-12T22:04:06.033 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:04:05 smithi097 podman[70558]: 2024-04-12 22:04:05.936145496 +0000 UTC m=+0.354476514 container start ed7a0de3252b5d661e49dfb254832ba1990db61bd9bb310a374b6d02e4a1a94e (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-0-activate, RELEASE=squid-b277feb, ceph=True, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-12T22:04:06.033 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:04:06 smithi097 podman[70558]: 2024-04-12 22:04:06.033270183 +0000 UTC m=+0.451601214 container attach ed7a0de3252b5d661e49dfb254832ba1990db61bd9bb310a374b6d02e4a1a94e (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-0-activate, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-b277feb, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-12T22:04:06.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:06 smithi097 ceph-mon[64853]: Health check failed: Degraded data redundancy: 67/630 objects degraded (10.635%), 20 pgs degraded (PG_DEGRADED) 2024-04-12T22:04:06.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:06 smithi097 ceph-mon[67407]: Health check failed: Degraded data redundancy: 67/630 objects degraded (10.635%), 20 pgs degraded (PG_DEGRADED) 2024-04-12T22:04:06.660 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:04:06 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-0-activate[70573]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-12T22:04:06.661 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:04:06 smithi097 bash[70558]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-12T22:04:06.661 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:04:06 smithi097 bash[70558]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-12T22:04:06.661 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:04:06 smithi097 bash[70558]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-12T22:04:06.661 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:04:06 smithi097 bash[70558]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-12T22:04:06.661 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:04:06 smithi097 bash[70558]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-04-12T22:04:06.661 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:04:06 smithi097 bash[70558]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-12T22:04:06.661 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:04:06 smithi097 bash[70558]: --> ceph-volume raw activate successful for osd ID: 0 2024-04-12T22:04:06.661 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:04:06 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-0-activate[70573]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-12T22:04:06.661 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:04:06 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-0-activate[70573]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-12T22:04:06.661 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:04:06 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-0-activate[70573]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-12T22:04:06.661 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:04:06 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-0-activate[70573]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-04-12T22:04:06.662 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:04:06 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-0-activate[70573]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-12T22:04:06.662 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:04:06 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-0-activate[70573]: --> ceph-volume raw activate successful for osd ID: 0 2024-04-12T22:04:06.662 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:04:06 smithi097 podman[70558]: 2024-04-12 22:04:06.401985597 +0000 UTC m=+0.820316621 container died ed7a0de3252b5d661e49dfb254832ba1990db61bd9bb310a374b6d02e4a1a94e (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-0-activate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b277feb) 2024-04-12T22:04:06.828 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:06 smithi114 ceph-mon[58780]: Health check failed: Degraded data redundancy: 67/630 objects degraded (10.635%), 20 pgs degraded (PG_DEGRADED) 2024-04-12T22:04:07.342 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:04:07 smithi097 podman[70694]: 2024-04-12 22:04:07.003968289 +0000 UTC m=+0.594819829 container remove ed7a0de3252b5d661e49dfb254832ba1990db61bd9bb310a374b6d02e4a1a94e (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-0-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-b277feb, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409) 2024-04-12T22:04:07.343 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:04:07 smithi097 podman[70757]: 2024-04-12 22:04:07.213012333 +0000 UTC m=+0.115415282 container create 9d05fea4bf37616fb1adc9115b72fb0b082b25b09f686f5d1b5293fdfb078675 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-b277feb, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-12T22:04:07.343 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:04:07 smithi097 podman[70757]: 2024-04-12 22:04:07.116362472 +0000 UTC m=+0.018765435 image pull 668895944f0ec64b67a5b7564961d2756f2966e5d4b9338516300dbec9984951 quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a 2024-04-12T22:04:07.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:07 smithi097 ceph-mon[64853]: pgmap v51: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 67/630 objects degraded (10.635%) 2024-04-12T22:04:07.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:07 smithi097 ceph-mon[67407]: pgmap v51: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 67/630 objects degraded (10.635%) 2024-04-12T22:04:07.661 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:04:07 smithi097 podman[70757]: 2024-04-12 22:04:07.432091154 +0000 UTC m=+0.334494107 container init 9d05fea4bf37616fb1adc9115b72fb0b082b25b09f686f5d1b5293fdfb078675 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-0, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-b277feb, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-12T22:04:07.661 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:04:07 smithi097 podman[70757]: 2024-04-12 22:04:07.435701898 +0000 UTC m=+0.338104847 container start 9d05fea4bf37616fb1adc9115b72fb0b082b25b09f686f5d1b5293fdfb078675 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-0, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=squid-b277feb, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-12T22:04:07.661 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:04:07 smithi097 ceph-osd[70776]: -- 172.21.15.97:0/1360058367 <== mon.1 v2:172.21.15.97:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5625555b7680 con 0x562555581400 2024-04-12T22:04:07.661 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:04:07 smithi097 bash[70757]: 9d05fea4bf37616fb1adc9115b72fb0b082b25b09f686f5d1b5293fdfb078675 2024-04-12T22:04:07.661 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:04:07 smithi097 systemd[1]: Started Ceph osd.0 for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:04:07.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:07 smithi114 ceph-mon[58780]: pgmap v51: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 67/630 objects degraded (10.635%) 2024-04-12T22:04:08.410 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:04:07 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-0[70772]: 2024-04-12T22:04:07.973+0000 7fd9c7b8e7c0 -1 Falling back to public interface 2024-04-12T22:04:08.910 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:08 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:08.910 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:08 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:08.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:08 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:08.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:08 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:08.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:08 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:08.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:08 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:08.993 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:04:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:04:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"smithi097:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"172.21.15.114:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-12T22:04:09.608 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:04:09 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:04:09] "GET /metrics HTTP/1.1" 200 38267 "" "Prometheus/2.33.4" 2024-04-12T22:04:09.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:09 smithi097 ceph-mon[64853]: pgmap v52: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 67/630 objects degraded (10.635%) 2024-04-12T22:04:09.910 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:09 smithi097 ceph-mon[67407]: pgmap v52: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 67/630 objects degraded (10.635%) 2024-04-12T22:04:09.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:09 smithi114 ceph-mon[58780]: pgmap v52: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 67/630 objects degraded (10.635%) 2024-04-12T22:04:10.967 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:04:10 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-0[70772]: 2024-04-12T22:04:10.699+0000 7fd9c7b8e7c0 -1 osd.0 0 read_superblock omap replica is missing. 2024-04-12T22:04:10.967 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:04:10 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-0[70772]: 2024-04-12T22:04:10.732+0000 7fd9c7b8e7c0 -1 osd.0 106 log_to_monitors true 2024-04-12T22:04:10.967 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:10 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:10.968 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:10 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:04:10.968 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:10 smithi097 ceph-mon[64853]: from='osd.0 [v2:172.21.15.97:6802/621540193,v1:172.21.15.97:6803/621540193]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-12T22:04:10.968 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:10 smithi097 ceph-mon[64853]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-12T22:04:10.968 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:10 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:10.968 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:10 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:04:10.968 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:10 smithi097 ceph-mon[67407]: from='osd.0 [v2:172.21.15.97:6802/621540193,v1:172.21.15.97:6803/621540193]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-12T22:04:10.968 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:10 smithi097 ceph-mon[67407]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-12T22:04:11.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:10 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:11.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:10 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:04:11.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:10 smithi114 ceph-mon[58780]: from='osd.0 [v2:172.21.15.97:6802/621540193,v1:172.21.15.97:6803/621540193]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-12T22:04:11.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:10 smithi114 ceph-mon[58780]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-12T22:04:11.660 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:04:11 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-0[70772]: 2024-04-12T22:04:11.223+0000 7fd9b4fde700 -1 osd.0 106 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T22:04:12.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:11 smithi097 ceph-mon[64853]: pgmap v53: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 67/630 objects degraded (10.635%) 2024-04-12T22:04:12.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:11 smithi097 ceph-mon[64853]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-12T22:04:12.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:11 smithi097 ceph-mon[64853]: osdmap e109: 8 total, 7 up, 8 in 2024-04-12T22:04:12.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:11 smithi097 ceph-mon[64853]: from='osd.0 [v2:172.21.15.97:6802/621540193,v1:172.21.15.97:6803/621540193]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-12T22:04:12.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:11 smithi097 ceph-mon[64853]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-12T22:04:12.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:11 smithi097 ceph-mon[64853]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:04:12.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:11 smithi097 ceph-mon[64853]: from='osd.0 ' entity='osd.0' 2024-04-12T22:04:12.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:11 smithi097 ceph-mon[67407]: pgmap v53: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 67/630 objects degraded (10.635%) 2024-04-12T22:04:12.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:11 smithi097 ceph-mon[67407]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-12T22:04:12.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:11 smithi097 ceph-mon[67407]: osdmap e109: 8 total, 7 up, 8 in 2024-04-12T22:04:12.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:11 smithi097 ceph-mon[67407]: from='osd.0 [v2:172.21.15.97:6802/621540193,v1:172.21.15.97:6803/621540193]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-12T22:04:12.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:11 smithi097 ceph-mon[67407]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-12T22:04:12.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:11 smithi097 ceph-mon[67407]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:04:12.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:11 smithi097 ceph-mon[67407]: from='osd.0 ' entity='osd.0' 2024-04-12T22:04:12.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:11 smithi114 ceph-mon[58780]: pgmap v53: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 67/630 objects degraded (10.635%) 2024-04-12T22:04:12.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:11 smithi114 ceph-mon[58780]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-12T22:04:12.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:11 smithi114 ceph-mon[58780]: osdmap e109: 8 total, 7 up, 8 in 2024-04-12T22:04:12.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:11 smithi114 ceph-mon[58780]: from='osd.0 [v2:172.21.15.97:6802/621540193,v1:172.21.15.97:6803/621540193]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-12T22:04:12.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:11 smithi114 ceph-mon[58780]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-12T22:04:12.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:11 smithi114 ceph-mon[58780]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:04:12.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:11 smithi114 ceph-mon[58780]: from='osd.0 ' entity='osd.0' 2024-04-12T22:04:13.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:13 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:13.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:13 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:13.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:13 smithi097 ceph-mon[64853]: pgmap v55: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 67/630 objects degraded (10.635%) 2024-04-12T22:04:13.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:13 smithi097 ceph-mon[64853]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-12T22:04:13.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:13 smithi097 ceph-mon[64853]: osd.0 [v2:172.21.15.97:6802/621540193,v1:172.21.15.97:6803/621540193] boot 2024-04-12T22:04:13.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:13 smithi097 ceph-mon[64853]: osdmap e110: 8 total, 8 up, 8 in 2024-04-12T22:04:13.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:13 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T22:04:13.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:13 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:13.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:13 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:13.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:13 smithi097 ceph-mon[67407]: pgmap v55: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 67/630 objects degraded (10.635%) 2024-04-12T22:04:13.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:13 smithi097 ceph-mon[67407]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-12T22:04:13.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:13 smithi097 ceph-mon[67407]: osd.0 [v2:172.21.15.97:6802/621540193,v1:172.21.15.97:6803/621540193] boot 2024-04-12T22:04:13.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:13 smithi097 ceph-mon[67407]: osdmap e110: 8 total, 8 up, 8 in 2024-04-12T22:04:13.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:13 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T22:04:13.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:13 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:13.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:13 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:13.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:13 smithi114 ceph-mon[58780]: pgmap v55: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 67/630 objects degraded (10.635%) 2024-04-12T22:04:13.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:13 smithi114 ceph-mon[58780]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-12T22:04:13.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:13 smithi114 ceph-mon[58780]: osd.0 [v2:172.21.15.97:6802/621540193,v1:172.21.15.97:6803/621540193] boot 2024-04-12T22:04:13.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:13 smithi114 ceph-mon[58780]: osdmap e110: 8 total, 8 up, 8 in 2024-04-12T22:04:13.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:13 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T22:04:14.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:14 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:14.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:14 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:14.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:14 smithi097 ceph-mon[64853]: osdmap e111: 8 total, 8 up, 8 in 2024-04-12T22:04:14.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:14 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:14.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:14 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:14.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:14 smithi097 ceph-mon[67407]: osdmap e111: 8 total, 8 up, 8 in 2024-04-12T22:04:14.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:14 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:14.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:14 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:14.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:14 smithi114 ceph-mon[58780]: osdmap e111: 8 total, 8 up, 8 in 2024-04-12T22:04:14.493 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:04:13 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@grafana.a.service: Scheduled restart job, restart counter is at 14. 2024-04-12T22:04:14.493 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:04:13 smithi114 systemd[1]: Stopped Ceph grafana.a for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:04:14.494 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:04:13 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@grafana.a.service: Start request repeated too quickly. 2024-04-12T22:04:14.494 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:04:13 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-04-12T22:04:14.494 INFO:journalctl@ceph.grafana.a.smithi114.stdout:Apr 12 22:04:13 smithi114 systemd[1]: Failed to start Ceph grafana.a for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:04:15.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:15 smithi097 ceph-mon[64853]: pgmap v58: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 67/630 objects degraded (10.635%) 2024-04-12T22:04:15.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:15 smithi097 ceph-mon[67407]: pgmap v58: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 67/630 objects degraded (10.635%) 2024-04-12T22:04:15.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:15 smithi114 ceph-mon[58780]: pgmap v58: 161 pgs: 20 active+undersized+degraded, 38 active+undersized, 103 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 67/630 objects degraded (10.635%) 2024-04-12T22:04:16.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:16 smithi097 ceph-mon[64853]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 67/630 objects degraded (10.635%), 20 pgs degraded) 2024-04-12T22:04:16.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:16 smithi097 ceph-mon[67407]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 67/630 objects degraded (10.635%), 20 pgs degraded) 2024-04-12T22:04:16.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:16 smithi114 ceph-mon[58780]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 67/630 objects degraded (10.635%), 20 pgs degraded) 2024-04-12T22:04:17.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:17 smithi114 ceph-mon[58780]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-12T22:04:17.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:17 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:17.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:17 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:17.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:17 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:04:17.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:17 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:04:17.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:17 smithi114 ceph-mon[58780]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 855 B/s rd, 0 op/s 2024-04-12T22:04:17.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:17 smithi114 ceph-mon[58780]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-12T22:04:17.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:17 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:17.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:17 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:04:17.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:17 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:04:17.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:17 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:04:17.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:17 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:04:17.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:17 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-12T22:04:18.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:17 smithi097 ceph-mon[64853]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-12T22:04:18.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:17 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:18.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:17 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:18.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:17 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:04:18.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:17 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:04:18.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:17 smithi097 ceph-mon[64853]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 855 B/s rd, 0 op/s 2024-04-12T22:04:18.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:17 smithi097 ceph-mon[64853]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-12T22:04:18.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:17 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:18.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:17 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:04:18.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:17 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:04:18.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:17 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:04:18.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:17 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:04:18.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:17 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-12T22:04:18.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:17 smithi097 ceph-mon[67407]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-12T22:04:18.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:17 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:18.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:17 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:18.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:17 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:04:18.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:17 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:04:18.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:17 smithi097 ceph-mon[67407]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 855 B/s rd, 0 op/s 2024-04-12T22:04:18.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:17 smithi097 ceph-mon[67407]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-12T22:04:18.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:17 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:18.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:17 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:04:18.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:17 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:04:18.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:17 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:04:18.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:17 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:04:18.163 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:17 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-12T22:04:18.993 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:04:18 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:04:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"smithi097:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"172.21.15.114:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-12T22:04:18.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:18 smithi114 ceph-mon[58780]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-12T22:04:18.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:18 smithi114 ceph-mon[58780]: Upgrade: osd.1 is safe to restart 2024-04-12T22:04:18.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:18 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:18.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:18 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-12T22:04:18.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:18 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:04:19.155 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:18 smithi097 ceph-mon[64853]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-12T22:04:19.155 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:18 smithi097 ceph-mon[64853]: Upgrade: osd.1 is safe to restart 2024-04-12T22:04:19.155 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:18 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:19.155 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:18 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-12T22:04:19.155 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:18 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:04:19.155 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:18 smithi097 ceph-mon[67407]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-12T22:04:19.156 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:18 smithi097 ceph-mon[67407]: Upgrade: osd.1 is safe to restart 2024-04-12T22:04:19.156 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:18 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:19.156 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:18 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-12T22:04:19.156 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:18 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:04:19.410 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:04:19 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:04:19] "GET /metrics HTTP/1.1" 200 38401 "" "Prometheus/2.33.4" 2024-04-12T22:04:20.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:19 smithi097 ceph-mon[64853]: Upgrade: Updating osd.1 2024-04-12T22:04:20.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:19 smithi097 ceph-mon[64853]: Deploying daemon osd.1 on smithi097 2024-04-12T22:04:20.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:19 smithi097 ceph-mon[64853]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 784 B/s rd, 0 op/s 2024-04-12T22:04:20.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:19 smithi097 ceph-mon[67407]: Upgrade: Updating osd.1 2024-04-12T22:04:20.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:19 smithi097 ceph-mon[67407]: Deploying daemon osd.1 on smithi097 2024-04-12T22:04:20.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:19 smithi097 ceph-mon[67407]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 784 B/s rd, 0 op/s 2024-04-12T22:04:20.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:19 smithi114 ceph-mon[58780]: Upgrade: Updating osd.1 2024-04-12T22:04:20.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:19 smithi114 ceph-mon[58780]: Deploying daemon osd.1 on smithi097 2024-04-12T22:04:20.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:19 smithi114 ceph-mon[58780]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 784 B/s rd, 0 op/s 2024-04-12T22:04:21.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:21 smithi097 ceph-mon[64853]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-12T22:04:21.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:21 smithi097 ceph-mon[64853]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:04:21.910 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:21 smithi097 ceph-mon[67407]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-12T22:04:21.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:21 smithi097 ceph-mon[67407]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:04:22.242 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:04:21 smithi097 systemd[1]: Stopping Ceph osd.1 for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:04:22.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:21 smithi114 ceph-mon[58780]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-12T22:04:22.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:21 smithi114 ceph-mon[58780]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:04:22.660 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:04:22 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-1[35441]: 2024-04-12T22:04:22.241+0000 7fdc2f1fd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-12T22:04:22.660 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:04:22 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-1[35441]: 2024-04-12T22:04:22.241+0000 7fdc2f1fd700 -1 osd.1 111 *** Got signal Terminated *** 2024-04-12T22:04:22.660 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:04:22 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-1[35441]: 2024-04-12T22:04:22.241+0000 7fdc2f1fd700 -1 osd.1 111 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-12T22:04:23.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:22 smithi097 ceph-mon[64853]: osd.1 marked itself down and dead 2024-04-12T22:04:23.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:22 smithi097 ceph-mon[67407]: osd.1 marked itself down and dead 2024-04-12T22:04:23.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:22 smithi114 ceph-mon[58780]: osd.1 marked itself down and dead 2024-04-12T22:04:23.577 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:04:23 smithi097 podman[72641]: 2024-04-12 22:04:23.259965054 +0000 UTC m=+1.139768143 container died 39192562d598b661db36af2c1857be3635955552a80276bef5cc830f67c43225 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-1, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, version=8, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, release=754, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git) 2024-04-12T22:04:23.902 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:23 smithi097 ceph-mon[64853]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:04:23.902 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:23 smithi097 ceph-mon[64853]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-12T22:04:23.902 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:23 smithi097 ceph-mon[64853]: osdmap e112: 8 total, 7 up, 8 in 2024-04-12T22:04:23.902 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:23 smithi097 ceph-mon[67407]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:04:23.902 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:23 smithi097 ceph-mon[67407]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-12T22:04:23.902 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:23 smithi097 ceph-mon[67407]: osdmap e112: 8 total, 7 up, 8 in 2024-04-12T22:04:23.903 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:04:23 smithi097 podman[72641]: 2024-04-12 22:04:23.675218791 +0000 UTC m=+1.555021858 container cleanup 39192562d598b661db36af2c1857be3635955552a80276bef5cc830f67c43225 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-1, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , distribution-scope=public, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, vendor=Red Hat, Inc., GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, ceph=True, io.buildah.version=1.19.8, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, name=centos-stream, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-12T22:04:23.903 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:04:23 smithi097 bash[72641]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-1 2024-04-12T22:04:24.160 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:04:23 smithi097 podman[72658]: 2024-04-12 22:04:23.901568091 +0000 UTC m=+0.636912678 container remove 39192562d598b661db36af2c1857be3635955552a80276bef5cc830f67c43225 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-1, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, architecture=x86_64, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, distribution-scope=public, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, ceph=True) 2024-04-12T22:04:24.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:23 smithi114 ceph-mon[58780]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:04:24.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:23 smithi114 ceph-mon[58780]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-12T22:04:24.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:23 smithi114 ceph-mon[58780]: osdmap e112: 8 total, 7 up, 8 in 2024-04-12T22:04:24.655 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:04:24 smithi097 podman[72750]: 2024-04-12 22:04:24.247506529 +0000 UTC m=+0.017528201 image pull 668895944f0ec64b67a5b7564961d2756f2966e5d4b9338516300dbec9984951 quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a 2024-04-12T22:04:24.655 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:04:24 smithi097 podman[72750]: 2024-04-12 22:04:24.352592855 +0000 UTC m=+0.122614520 container create 66b73628d98156704833064a1d8cd16f01ee3dadc922f90227ba79e09aebcf7f (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-1-deactivate, RELEASE=squid-b277feb, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-12T22:04:24.655 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:04:24 smithi097 podman[72750]: 2024-04-12 22:04:24.571879136 +0000 UTC m=+0.341900798 container init 66b73628d98156704833064a1d8cd16f01ee3dadc922f90227ba79e09aebcf7f (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-1-deactivate, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-b277feb, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True) 2024-04-12T22:04:24.655 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:04:24 smithi097 podman[72750]: 2024-04-12 22:04:24.575180585 +0000 UTC m=+0.345202256 container start 66b73628d98156704833064a1d8cd16f01ee3dadc922f90227ba79e09aebcf7f (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240409, ceph=True, RELEASE=squid-b277feb, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-12T22:04:24.655 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:04:24 smithi097 podman[72750]: 2024-04-12 22:04:24.655013865 +0000 UTC m=+0.425035536 container attach 66b73628d98156704833064a1d8cd16f01ee3dadc922f90227ba79e09aebcf7f (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-1-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, RELEASE=squid-b277feb, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-12T22:04:24.910 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:04:24 smithi097 podman[72750]: 2024-04-12 22:04:24.879779817 +0000 UTC m=+0.649801487 container died 66b73628d98156704833064a1d8cd16f01ee3dadc922f90227ba79e09aebcf7f (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-b277feb) 2024-04-12T22:04:24.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:24 smithi097 ceph-mon[64853]: osdmap e113: 8 total, 7 up, 8 in 2024-04-12T22:04:24.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:24 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:24.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:24 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:04:24.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:24 smithi097 ceph-mon[67407]: osdmap e113: 8 total, 7 up, 8 in 2024-04-12T22:04:24.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:24 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:24.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:24 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:04:25.146 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:24 smithi114 ceph-mon[58780]: osdmap e113: 8 total, 7 up, 8 in 2024-04-12T22:04:25.146 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:24 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:25.146 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:24 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:04:25.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:25 smithi097 ceph-mon[64853]: pgmap v67: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-12T22:04:25.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:25 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:25.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:25 smithi097 ceph-mon[67407]: pgmap v67: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-12T22:04:25.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:25 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:25.911 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:04:25 smithi097 podman[72776]: 2024-04-12 22:04:25.608849212 +0000 UTC m=+0.718247659 container remove 66b73628d98156704833064a1d8cd16f01ee3dadc922f90227ba79e09aebcf7f (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-b277feb, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS) 2024-04-12T22:04:25.911 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:04:25 smithi097 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@osd.1.service: Deactivated successfully. 2024-04-12T22:04:25.912 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:04:25 smithi097 systemd[1]: Stopped Ceph osd.1 for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:04:25.912 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:04:25 smithi097 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@osd.1.service: Consumed 7.279s CPU time. 2024-04-12T22:04:25.912 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:04:25 smithi097 systemd[1]: Starting Ceph osd.1 for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:04:26.168 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:25 smithi114 ceph-mon[58780]: pgmap v67: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-12T22:04:26.168 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:25 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:26.354 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:04:26 smithi097 podman[72878]: 2024-04-12 22:04:25.960228981 +0000 UTC m=+0.018362664 image pull 668895944f0ec64b67a5b7564961d2756f2966e5d4b9338516300dbec9984951 quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a 2024-04-12T22:04:26.354 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:04:26 smithi097 podman[72878]: 2024-04-12 22:04:26.068203866 +0000 UTC m=+0.126337544 container create 690732dc11da8a3c378dc68b934b9dee89dce04b685dc34aa1dfc1223e9098a2 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-1-activate, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-b277feb, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-12T22:04:26.354 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:04:26 smithi097 podman[72878]: 2024-04-12 22:04:26.270998662 +0000 UTC m=+0.329132341 container init 690732dc11da8a3c378dc68b934b9dee89dce04b685dc34aa1dfc1223e9098a2 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-1-activate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-b277feb, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux ) 2024-04-12T22:04:26.354 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:04:26 smithi097 podman[72878]: 2024-04-12 22:04:26.274589806 +0000 UTC m=+0.332723493 container start 690732dc11da8a3c378dc68b934b9dee89dce04b685dc34aa1dfc1223e9098a2 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-1-activate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-b277feb, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-12T22:04:26.354 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:04:26 smithi097 podman[72878]: 2024-04-12 22:04:26.354237734 +0000 UTC m=+0.412371415 container attach 690732dc11da8a3c378dc68b934b9dee89dce04b685dc34aa1dfc1223e9098a2 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-1-activate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-b277feb, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2) 2024-04-12T22:04:27.071 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:26 smithi097 ceph-mon[64853]: Health check failed: Degraded data redundancy: 83/630 objects degraded (13.175%), 21 pgs degraded (PG_DEGRADED) 2024-04-12T22:04:27.071 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:26 smithi097 ceph-mon[67407]: Health check failed: Degraded data redundancy: 83/630 objects degraded (13.175%), 21 pgs degraded (PG_DEGRADED) 2024-04-12T22:04:27.071 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:04:26 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-1-activate[72893]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-12T22:04:27.071 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:04:26 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-1-activate[72893]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-12T22:04:27.071 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:04:26 smithi097 bash[72878]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-12T22:04:27.072 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:04:26 smithi097 bash[72878]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-12T22:04:27.072 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:04:26 smithi097 bash[72878]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-12T22:04:27.072 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:04:26 smithi097 bash[72878]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-12T22:04:27.072 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:04:26 smithi097 bash[72878]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-04-12T22:04:27.072 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:04:26 smithi097 bash[72878]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-12T22:04:27.072 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:04:26 smithi097 bash[72878]: --> ceph-volume raw activate successful for osd ID: 1 2024-04-12T22:04:27.072 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:04:26 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-1-activate[72893]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-12T22:04:27.072 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:04:26 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-1-activate[72893]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-12T22:04:27.072 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:04:26 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-1-activate[72893]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-04-12T22:04:27.072 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:04:26 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-1-activate[72893]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-12T22:04:27.072 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:04:26 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-1-activate[72893]: --> ceph-volume raw activate successful for osd ID: 1 2024-04-12T22:04:27.072 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:04:26 smithi097 podman[72878]: 2024-04-12 22:04:26.738828985 +0000 UTC m=+0.796962664 container died 690732dc11da8a3c378dc68b934b9dee89dce04b685dc34aa1dfc1223e9098a2 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-1-activate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-b277feb, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409) 2024-04-12T22:04:27.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:26 smithi114 ceph-mon[58780]: Health check failed: Degraded data redundancy: 83/630 objects degraded (13.175%), 21 pgs degraded (PG_DEGRADED) 2024-04-12T22:04:27.660 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:04:27 smithi097 podman[73014]: 2024-04-12 22:04:27.333399504 +0000 UTC m=+0.589196020 container remove 690732dc11da8a3c378dc68b934b9dee89dce04b685dc34aa1dfc1223e9098a2 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-1-activate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, RELEASE=squid-b277feb, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2) 2024-04-12T22:04:27.660 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:04:27 smithi097 podman[73077]: 2024-04-12 22:04:27.447864855 +0000 UTC m=+0.017811656 image pull 668895944f0ec64b67a5b7564961d2756f2966e5d4b9338516300dbec9984951 quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a 2024-04-12T22:04:27.660 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:04:27 smithi097 podman[73077]: 2024-04-12 22:04:27.559137075 +0000 UTC m=+0.129083851 container create a9a527642889e19d8f67c54d289d38bd5e436000c65d53291c8354c2eeb12c0a (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-1, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-b277feb) 2024-04-12T22:04:28.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:27 smithi097 ceph-mon[64853]: pgmap v68: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 83/630 objects degraded (13.175%) 2024-04-12T22:04:28.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:27 smithi097 ceph-mon[67407]: pgmap v68: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 83/630 objects degraded (13.175%) 2024-04-12T22:04:28.161 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:04:27 smithi097 podman[73077]: 2024-04-12 22:04:27.761393278 +0000 UTC m=+0.331340055 container init a9a527642889e19d8f67c54d289d38bd5e436000c65d53291c8354c2eeb12c0a (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-1, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, RELEASE=squid-b277feb, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-12T22:04:28.161 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:04:27 smithi097 podman[73077]: 2024-04-12 22:04:27.764871033 +0000 UTC m=+0.334817807 container start a9a527642889e19d8f67c54d289d38bd5e436000c65d53291c8354c2eeb12c0a (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-1, ceph=True, org.label-schema.build-date=20240409, RELEASE=squid-b277feb, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-12T22:04:28.161 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:04:27 smithi097 ceph-osd[73097]: -- 172.21.15.97:0/61766117 <== mon.1 v2:172.21.15.97:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55bbaae7d680 con 0x55bbaae47000 2024-04-12T22:04:28.161 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:04:27 smithi097 bash[73077]: a9a527642889e19d8f67c54d289d38bd5e436000c65d53291c8354c2eeb12c0a 2024-04-12T22:04:28.161 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:04:27 smithi097 systemd[1]: Started Ceph osd.1 for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:04:28.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:27 smithi114 ceph-mon[58780]: pgmap v68: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 83/630 objects degraded (13.175%) 2024-04-12T22:04:28.660 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:04:28 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-1[73093]: 2024-04-12T22:04:28.306+0000 7febdc43c7c0 -1 Falling back to public interface 2024-04-12T22:04:28.914 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:04:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:04:28.659Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"smithi097:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"172.21.15.114:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-04-12T22:04:28.914 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:04:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:04:28.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"smithi097:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"172.21.15.114:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-12T22:04:29.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:28 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:29.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:28 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:29.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:28 smithi097 ceph-mon[64853]: pgmap v69: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 83/630 objects degraded (13.175%) 2024-04-12T22:04:29.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:28 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:29.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:28 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:29.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:28 smithi097 ceph-mon[67407]: pgmap v69: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 83/630 objects degraded (13.175%) 2024-04-12T22:04:29.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:28 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:29.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:28 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:29.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:28 smithi114 ceph-mon[58780]: pgmap v69: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 83/630 objects degraded (13.175%) 2024-04-12T22:04:29.290 INFO:teuthology.orchestra.run.smithi097.stdout:true 2024-04-12T22:04:29.660 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:04:29 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:04:29] "GET /metrics HTTP/1.1" 200 38409 "" "Prometheus/2.33.4" 2024-04-12T22:04:29.968 INFO:teuthology.orchestra.run.smithi097.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T22:04:29.969 INFO:teuthology.orchestra.run.smithi097.stdout:alertmanager.a smithi097 *:9093,9094 running (3m) 17s ago 8m 26.3M - 0.23.0 ba2b418f427c 7b4c491fe0a0 2024-04-12T22:04:29.969 INFO:teuthology.orchestra.run.smithi097.stdout:grafana.a smithi114 *:3000 unknown 67s ago 8m - - 2024-04-12T22:04:29.969 INFO:teuthology.orchestra.run.smithi097.stdout:iscsi.foo.smithi097.xndkxy smithi097 running (2m) 17s ago 7m 50.1M - 3.5 e1d6a67b021e 0436463f28cd 2024-04-12T22:04:29.969 INFO:teuthology.orchestra.run.smithi097.stdout:mgr.x smithi114 *:8443,9283,8765 running (2m) 67s ago 12m 450M - 19.0.0-2053-gb277feb2 668895944f0e b8f8df3bf928 2024-04-12T22:04:29.969 INFO:teuthology.orchestra.run.smithi097.stdout:mgr.y smithi097 *:8443,9283,8765 running (2m) 17s ago 14m 502M - 19.0.0-2053-gb277feb2 668895944f0e 488170dac11f 2024-04-12T22:04:29.969 INFO:teuthology.orchestra.run.smithi097.stdout:mon.a smithi097 running (2m) 17s ago 14m 48.0M 2048M 19.0.0-2053-gb277feb2 668895944f0e 3cfa16d3a85c 2024-04-12T22:04:29.969 INFO:teuthology.orchestra.run.smithi097.stdout:mon.b smithi114 running (78s) 67s ago 13m 18.4M 2048M 19.0.0-2053-gb277feb2 668895944f0e 71a2eff159a3 2024-04-12T22:04:29.970 INFO:teuthology.orchestra.run.smithi097.stdout:mon.c smithi097 running (94s) 17s ago 13m 37.1M 2048M 19.0.0-2053-gb277feb2 668895944f0e 69507ba88603 2024-04-12T22:04:29.970 INFO:teuthology.orchestra.run.smithi097.stdout:node-exporter.a smithi097 *:9100 running (3m) 17s ago 9m 18.1M - 1.3.1 1dbe0e931976 5edff9053b37 2024-04-12T22:04:29.970 INFO:teuthology.orchestra.run.smithi097.stdout:node-exporter.b smithi114 *:9100 running (3m) 67s ago 9m 17.0M - 1.3.1 1dbe0e931976 904afc45dc2f 2024-04-12T22:04:29.970 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0 smithi097 running (22s) 17s ago 12m 12.7M 2736M 19.0.0-2053-gb277feb2 668895944f0e 9d05fea4bf37 2024-04-12T22:04:29.970 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1 smithi097 starting - - - 2736M 2024-04-12T22:04:29.970 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2 smithi097 running (11m) 17s ago 11m 52.4M 2736M 17.2.0 e1d6a67b021e 89daf8e339c3 2024-04-12T22:04:29.970 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3 smithi097 running (11m) 17s ago 11m 53.0M 2736M 17.2.0 e1d6a67b021e 4e9a0b3862d9 2024-04-12T22:04:29.970 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4 smithi114 running (10m) 67s ago 10m 53.6M 2992M 17.2.0 e1d6a67b021e fb42c0e8e575 2024-04-12T22:04:29.970 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5 smithi114 running (10m) 67s ago 10m 50.2M 2992M 17.2.0 e1d6a67b021e 3acd4a3b61e5 2024-04-12T22:04:29.970 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6 smithi114 running (10m) 67s ago 10m 49.6M 2992M 17.2.0 e1d6a67b021e 5969baf67f41 2024-04-12T22:04:29.970 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7 smithi114 running (9m) 67s ago 9m 51.8M 2992M 17.2.0 e1d6a67b021e f68340fecf2c 2024-04-12T22:04:29.970 INFO:teuthology.orchestra.run.smithi097.stdout:prometheus.a smithi114 *:9095 running (2m) 67s ago 9m 65.1M - 2.33.4 514e6a882f6e 0ad5437ed63f 2024-04-12T22:04:29.970 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.foo.smithi097.sggtkh smithi097 *:8000 running (7m) 17s ago 7m 72.6M - 17.2.0 e1d6a67b021e a39f9b2d8509 2024-04-12T22:04:29.971 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.foo.smithi114.ulfbyo smithi114 *:8000 running (7m) 67s ago 7m 72.8M - 17.2.0 e1d6a67b021e df79a48bdd59 2024-04-12T22:04:29.971 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.smpl.smithi097.exxkgl smithi097 *:80 running (7m) 17s ago 7m 72.6M - 17.2.0 e1d6a67b021e a036c58a0099 2024-04-12T22:04:29.971 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.smpl.smithi114.lrgvph smithi114 *:80 running (7m) 67s ago 7m 74.4M - 17.2.0 e1d6a67b021e 031a5fc24237 2024-04-12T22:04:30.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:29 smithi114 ceph-mon[58780]: from='client.34251 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:04:30.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:29 smithi114 ceph-mon[58780]: from='client.34257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:04:30.384 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-12T22:04:30.384 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": { 2024-04-12T22:04:30.384 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 3 2024-04-12T22:04:30.384 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:04:30.384 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": { 2024-04-12T22:04:30.385 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 2 2024-04-12T22:04:30.385 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:04:30.385 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": { 2024-04-12T22:04:30.385 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-04-12T22:04:30.385 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 1 2024-04-12T22:04:30.385 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:04:30.385 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw": { 2024-04-12T22:04:30.385 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-12T22:04:30.385 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:04:30.385 INFO:teuthology.orchestra.run.smithi097.stdout: "overall": { 2024-04-12T22:04:30.385 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-04-12T22:04:30.385 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 6 2024-04-12T22:04:30.385 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-12T22:04:30.385 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-12T22:04:30.386 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:29 smithi097 ceph-mon[64853]: from='client.34251 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:04:30.386 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:29 smithi097 ceph-mon[64853]: from='client.34257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:04:30.386 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:29 smithi097 ceph-mon[67407]: from='client.34251 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:04:30.386 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:29 smithi097 ceph-mon[67407]: from='client.34257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:04:30.720 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-12T22:04:30.720 INFO:teuthology.orchestra.run.smithi097.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a", 2024-04-12T22:04:30.721 INFO:teuthology.orchestra.run.smithi097.stdout: "in_progress": true, 2024-04-12T22:04:30.721 INFO:teuthology.orchestra.run.smithi097.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T22:04:30.721 INFO:teuthology.orchestra.run.smithi097.stdout: "services_complete": [ 2024-04-12T22:04:30.721 INFO:teuthology.orchestra.run.smithi097.stdout: "mon", 2024-04-12T22:04:30.721 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr" 2024-04-12T22:04:30.721 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-04-12T22:04:30.721 INFO:teuthology.orchestra.run.smithi097.stdout: "progress": "6/23 daemons upgraded", 2024-04-12T22:04:30.721 INFO:teuthology.orchestra.run.smithi097.stdout: "message": "Currently upgrading osd daemons", 2024-04-12T22:04:30.721 INFO:teuthology.orchestra.run.smithi097.stdout: "is_paused": false 2024-04-12T22:04:30.721 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-12T22:04:31.140 INFO:teuthology.orchestra.run.smithi097.stdout:HEALTH_WARN 1 failed cephadm daemon(s); 1 osds down; Degraded data redundancy: 83/630 objects degraded (13.175%), 21 pgs degraded 2024-04-12T22:04:31.140 INFO:teuthology.orchestra.run.smithi097.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-12T22:04:31.141 INFO:teuthology.orchestra.run.smithi097.stdout: daemon grafana.a on smithi114 is in unknown state 2024-04-12T22:04:31.141 INFO:teuthology.orchestra.run.smithi097.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-12T22:04:31.141 INFO:teuthology.orchestra.run.smithi097.stdout: osd.1 (root=default,host=smithi097) is down 2024-04-12T22:04:31.141 INFO:teuthology.orchestra.run.smithi097.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 83/630 objects degraded (13.175%), 21 pgs degraded 2024-04-12T22:04:31.141 INFO:teuthology.orchestra.run.smithi097.stdout: pg 2.0 is active+undersized+degraded, acting [2,0] 2024-04-12T22:04:31.141 INFO:teuthology.orchestra.run.smithi097.stdout: pg 2.9 is active+undersized+degraded, acting [7,3] 2024-04-12T22:04:31.141 INFO:teuthology.orchestra.run.smithi097.stdout: pg 2.a is active+undersized+degraded, acting [3,7] 2024-04-12T22:04:31.141 INFO:teuthology.orchestra.run.smithi097.stdout: pg 3.0 is active+undersized+degraded, acting [2,6] 2024-04-12T22:04:31.141 INFO:teuthology.orchestra.run.smithi097.stdout: pg 3.4 is active+undersized+degraded, acting [2,5] 2024-04-12T22:04:31.141 INFO:teuthology.orchestra.run.smithi097.stdout: pg 3.6 is active+undersized+degraded, acting [0,4] 2024-04-12T22:04:31.141 INFO:teuthology.orchestra.run.smithi097.stdout: pg 3.8 is active+undersized+degraded, acting [3,7] 2024-04-12T22:04:31.141 INFO:teuthology.orchestra.run.smithi097.stdout: pg 3.a is active+undersized+degraded, acting [6,4] 2024-04-12T22:04:31.141 INFO:teuthology.orchestra.run.smithi097.stdout: pg 3.e is active+undersized+degraded, acting [7,4] 2024-04-12T22:04:31.141 INFO:teuthology.orchestra.run.smithi097.stdout: pg 3.16 is active+undersized+degraded, acting [5,7] 2024-04-12T22:04:31.141 INFO:teuthology.orchestra.run.smithi097.stdout: pg 3.18 is active+undersized+degraded, acting [3,0] 2024-04-12T22:04:31.142 INFO:teuthology.orchestra.run.smithi097.stdout: pg 3.19 is active+undersized+degraded, acting [3,4] 2024-04-12T22:04:31.142 INFO:teuthology.orchestra.run.smithi097.stdout: pg 3.1a is active+undersized+degraded, acting [4,2] 2024-04-12T22:04:31.142 INFO:teuthology.orchestra.run.smithi097.stdout: pg 3.1c is active+undersized+degraded, acting [5,4] 2024-04-12T22:04:31.142 INFO:teuthology.orchestra.run.smithi097.stdout: pg 4.9 is active+undersized+degraded, acting [4,3] 2024-04-12T22:04:31.142 INFO:teuthology.orchestra.run.smithi097.stdout: pg 4.d is active+undersized+degraded, acting [4,2] 2024-04-12T22:04:31.142 INFO:teuthology.orchestra.run.smithi097.stdout: pg 4.e is active+undersized+degraded, acting [4,6] 2024-04-12T22:04:31.142 INFO:teuthology.orchestra.run.smithi097.stdout: pg 4.14 is active+undersized+degraded, acting [3,7] 2024-04-12T22:04:31.142 INFO:teuthology.orchestra.run.smithi097.stdout: pg 4.1f is active+undersized+degraded, acting [6,5] 2024-04-12T22:04:31.142 INFO:teuthology.orchestra.run.smithi097.stdout: pg 5.1d is active+undersized+degraded, acting [4,0] 2024-04-12T22:04:31.142 INFO:teuthology.orchestra.run.smithi097.stdout: pg 6.1a is active+undersized+degraded, acting [4,5] 2024-04-12T22:04:31.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:30 smithi114 ceph-mon[58780]: from='client.34263 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:04:31.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:30 smithi114 ceph-mon[58780]: from='client.? 172.21.15.97:0/4188482685' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:04:31.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:30 smithi114 ceph-mon[58780]: from='client.44203 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:04:31.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:30 smithi114 ceph-mon[58780]: pgmap v70: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 83/630 objects degraded (13.175%) 2024-04-12T22:04:31.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:30 smithi114 ceph-mon[58780]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:04:31.304 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:04:31 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-1[73093]: 2024-04-12T22:04:31.020+0000 7febdc43c7c0 -1 osd.1 0 read_superblock omap replica is missing. 2024-04-12T22:04:31.304 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:04:31 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-1[73093]: 2024-04-12T22:04:31.047+0000 7febdc43c7c0 -1 osd.1 111 log_to_monitors true 2024-04-12T22:04:31.304 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:30 smithi097 ceph-mon[64853]: from='client.34263 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:04:31.304 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:30 smithi097 ceph-mon[64853]: from='client.? 172.21.15.97:0/4188482685' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:04:31.304 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:30 smithi097 ceph-mon[64853]: from='client.44203 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:04:31.304 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:30 smithi097 ceph-mon[64853]: pgmap v70: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 83/630 objects degraded (13.175%) 2024-04-12T22:04:31.304 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:30 smithi097 ceph-mon[64853]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:04:31.305 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:30 smithi097 ceph-mon[67407]: from='client.34263 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:04:31.305 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:30 smithi097 ceph-mon[67407]: from='client.? 172.21.15.97:0/4188482685' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:04:31.305 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:30 smithi097 ceph-mon[67407]: from='client.44203 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:04:31.305 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:30 smithi097 ceph-mon[67407]: pgmap v70: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 83/630 objects degraded (13.175%) 2024-04-12T22:04:31.305 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:30 smithi097 ceph-mon[67407]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:04:32.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:31 smithi114 ceph-mon[58780]: from='osd.1 [v2:172.21.15.97:6810/2058351284,v1:172.21.15.97:6811/2058351284]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-12T22:04:32.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:31 smithi114 ceph-mon[58780]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-12T22:04:32.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:31 smithi114 ceph-mon[58780]: from='client.? 172.21.15.97:0/2717435488' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T22:04:32.410 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:31 smithi097 ceph-mon[67407]: from='osd.1 [v2:172.21.15.97:6810/2058351284,v1:172.21.15.97:6811/2058351284]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-12T22:04:32.410 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:31 smithi097 ceph-mon[67407]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-12T22:04:32.410 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:31 smithi097 ceph-mon[67407]: from='client.? 172.21.15.97:0/2717435488' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T22:04:32.411 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:04:31 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-1[73093]: 2024-04-12T22:04:31.965+0000 7febcd894700 -1 osd.1 111 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T22:04:32.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:31 smithi097 ceph-mon[64853]: from='osd.1 [v2:172.21.15.97:6810/2058351284,v1:172.21.15.97:6811/2058351284]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-12T22:04:32.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:31 smithi097 ceph-mon[64853]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-12T22:04:32.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:31 smithi097 ceph-mon[64853]: from='client.? 172.21.15.97:0/2717435488' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T22:04:33.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:32 smithi114 ceph-mon[58780]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-12T22:04:33.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:32 smithi114 ceph-mon[58780]: osdmap e114: 8 total, 7 up, 8 in 2024-04-12T22:04:33.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:32 smithi114 ceph-mon[58780]: from='osd.1 [v2:172.21.15.97:6810/2058351284,v1:172.21.15.97:6811/2058351284]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-12T22:04:33.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:32 smithi114 ceph-mon[58780]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-12T22:04:33.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:32 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:33.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:32 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:33.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:32 smithi114 ceph-mon[58780]: pgmap v72: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 83/630 objects degraded (13.175%) 2024-04-12T22:04:33.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:32 smithi097 ceph-mon[64853]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-12T22:04:33.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:32 smithi097 ceph-mon[64853]: osdmap e114: 8 total, 7 up, 8 in 2024-04-12T22:04:33.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:32 smithi097 ceph-mon[64853]: from='osd.1 [v2:172.21.15.97:6810/2058351284,v1:172.21.15.97:6811/2058351284]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-12T22:04:33.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:32 smithi097 ceph-mon[64853]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-12T22:04:33.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:32 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:33.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:32 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:33.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:32 smithi097 ceph-mon[64853]: pgmap v72: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 83/630 objects degraded (13.175%) 2024-04-12T22:04:33.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:32 smithi097 ceph-mon[67407]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-12T22:04:33.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:32 smithi097 ceph-mon[67407]: osdmap e114: 8 total, 7 up, 8 in 2024-04-12T22:04:33.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:32 smithi097 ceph-mon[67407]: from='osd.1 [v2:172.21.15.97:6810/2058351284,v1:172.21.15.97:6811/2058351284]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-12T22:04:33.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:32 smithi097 ceph-mon[67407]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-12T22:04:33.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:32 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:33.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:32 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:33.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:32 smithi097 ceph-mon[67407]: pgmap v72: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 83/630 objects degraded (13.175%) 2024-04-12T22:04:34.219 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:33 smithi097 ceph-mon[64853]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-12T22:04:34.219 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:33 smithi097 ceph-mon[64853]: osd.1 [v2:172.21.15.97:6810/2058351284,v1:172.21.15.97:6811/2058351284] boot 2024-04-12T22:04:34.219 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:33 smithi097 ceph-mon[64853]: osdmap e115: 8 total, 8 up, 8 in 2024-04-12T22:04:34.220 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:33 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T22:04:34.220 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:33 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:34.220 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:33 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:34.220 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:33 smithi097 ceph-mon[67407]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-12T22:04:34.220 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:33 smithi097 ceph-mon[67407]: osd.1 [v2:172.21.15.97:6810/2058351284,v1:172.21.15.97:6811/2058351284] boot 2024-04-12T22:04:34.220 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:33 smithi097 ceph-mon[67407]: osdmap e115: 8 total, 8 up, 8 in 2024-04-12T22:04:34.220 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:33 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T22:04:34.220 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:33 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:34.220 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:33 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:34.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:33 smithi114 ceph-mon[58780]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-12T22:04:34.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:33 smithi114 ceph-mon[58780]: osd.1 [v2:172.21.15.97:6810/2058351284,v1:172.21.15.97:6811/2058351284] boot 2024-04-12T22:04:34.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:33 smithi114 ceph-mon[58780]: osdmap e115: 8 total, 8 up, 8 in 2024-04-12T22:04:34.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:33 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T22:04:34.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:33 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:34.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:33 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:35.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:34 smithi114 ceph-mon[58780]: osdmap e116: 8 total, 8 up, 8 in 2024-04-12T22:04:35.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:34 smithi114 ceph-mon[58780]: pgmap v75: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 83/630 objects degraded (13.175%) 2024-04-12T22:04:35.291 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:34 smithi097 ceph-mon[64853]: osdmap e116: 8 total, 8 up, 8 in 2024-04-12T22:04:35.291 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:34 smithi097 ceph-mon[64853]: pgmap v75: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 83/630 objects degraded (13.175%) 2024-04-12T22:04:35.291 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:34 smithi097 ceph-mon[67407]: osdmap e116: 8 total, 8 up, 8 in 2024-04-12T22:04:35.291 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:34 smithi097 ceph-mon[67407]: pgmap v75: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 130 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 83/630 objects degraded (13.175%) 2024-04-12T22:04:37.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:36 smithi097 ceph-mon[64853]: pgmap v76: 161 pgs: 6 active+undersized+degraded, 15 active+undersized, 140 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 29/630 objects degraded (4.603%) 2024-04-12T22:04:37.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:36 smithi097 ceph-mon[64853]: Health check update: Degraded data redundancy: 29/630 objects degraded (4.603%), 6 pgs degraded (PG_DEGRADED) 2024-04-12T22:04:37.160 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:04:37 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:04:37.096+0000 7fe40fa5a700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (5 PGs are or would become offline) 2024-04-12T22:04:37.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:36 smithi097 ceph-mon[67407]: pgmap v76: 161 pgs: 6 active+undersized+degraded, 15 active+undersized, 140 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 29/630 objects degraded (4.603%) 2024-04-12T22:04:37.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:36 smithi097 ceph-mon[67407]: Health check update: Degraded data redundancy: 29/630 objects degraded (4.603%), 6 pgs degraded (PG_DEGRADED) 2024-04-12T22:04:37.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:36 smithi114 ceph-mon[58780]: pgmap v76: 161 pgs: 6 active+undersized+degraded, 15 active+undersized, 140 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 29/630 objects degraded (4.603%) 2024-04-12T22:04:37.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:36 smithi114 ceph-mon[58780]: Health check update: Degraded data redundancy: 29/630 objects degraded (4.603%), 6 pgs degraded (PG_DEGRADED) 2024-04-12T22:04:38.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:37 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:38.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:37 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:38.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:37 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:04:38.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:37 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:04:38.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:37 smithi114 ceph-mon[58780]: pgmap v77: 161 pgs: 6 active+undersized+degraded, 15 active+undersized, 140 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s; 29/630 objects degraded (4.603%) 2024-04-12T22:04:38.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:37 smithi114 ceph-mon[58780]: pgmap v78: 161 pgs: 6 active+undersized+degraded, 15 active+undersized, 140 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 29/630 objects degraded (4.603%) 2024-04-12T22:04:38.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:37 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:38.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:37 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:04:38.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:37 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:04:38.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:37 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:04:38.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:37 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:04:38.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:37 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-12T22:04:38.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:37 smithi114 ceph-mon[58780]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-12T22:04:38.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:37 smithi114 ceph-mon[58780]: Upgrade: unsafe to stop osd(s) at this time (5 PGs are or would become offline) 2024-04-12T22:04:38.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:37 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:38.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:37 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:38.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:37 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:04:38.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:37 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:04:38.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:37 smithi097 ceph-mon[64853]: pgmap v77: 161 pgs: 6 active+undersized+degraded, 15 active+undersized, 140 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s; 29/630 objects degraded (4.603%) 2024-04-12T22:04:38.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:37 smithi097 ceph-mon[64853]: pgmap v78: 161 pgs: 6 active+undersized+degraded, 15 active+undersized, 140 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 29/630 objects degraded (4.603%) 2024-04-12T22:04:38.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:37 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:38.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:37 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:04:38.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:37 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:04:38.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:37 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:04:38.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:37 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:04:38.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:37 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-12T22:04:38.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:37 smithi097 ceph-mon[64853]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-12T22:04:38.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:37 smithi097 ceph-mon[64853]: Upgrade: unsafe to stop osd(s) at this time (5 PGs are or would become offline) 2024-04-12T22:04:38.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:37 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:38.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:37 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:38.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:37 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:04:38.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:37 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:04:38.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:37 smithi097 ceph-mon[67407]: pgmap v77: 161 pgs: 6 active+undersized+degraded, 15 active+undersized, 140 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s; 29/630 objects degraded (4.603%) 2024-04-12T22:04:38.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:37 smithi097 ceph-mon[67407]: pgmap v78: 161 pgs: 6 active+undersized+degraded, 15 active+undersized, 140 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 29/630 objects degraded (4.603%) 2024-04-12T22:04:38.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:37 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:38.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:37 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:04:38.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:37 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:04:38.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:37 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:04:38.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:37 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:04:38.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:37 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-12T22:04:38.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:37 smithi097 ceph-mon[67407]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-12T22:04:38.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:37 smithi097 ceph-mon[67407]: Upgrade: unsafe to stop osd(s) at this time (5 PGs are or would become offline) 2024-04-12T22:04:38.993 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:04:38 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:04:38.660Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"smithi097:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"172.21.15.114:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-04-12T22:04:38.993 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:04:38 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:04:38.661Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"smithi097:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"172.21.15.114:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-12T22:04:39.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:39 smithi097 ceph-mon[64853]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 29/630 objects degraded (4.603%), 6 pgs degraded) 2024-04-12T22:04:39.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:39 smithi097 ceph-mon[67407]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 29/630 objects degraded (4.603%), 6 pgs degraded) 2024-04-12T22:04:39.411 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:04:39 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:04:39] "GET /metrics HTTP/1.1" 200 38409 "" "Prometheus/2.33.4" 2024-04-12T22:04:39.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:39 smithi114 ceph-mon[58780]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 29/630 objects degraded (4.603%), 6 pgs degraded) 2024-04-12T22:04:41.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:40 smithi097 ceph-mon[64853]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-12T22:04:41.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:40 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:41.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:40 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:04:41.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:40 smithi097 ceph-mon[67407]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-12T22:04:41.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:40 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:41.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:40 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:04:41.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:40 smithi114 ceph-mon[58780]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-12T22:04:41.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:40 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:41.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:40 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:04:42.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:41 smithi097 ceph-mon[64853]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:04:42.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:41 smithi097 ceph-mon[67407]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:04:42.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:41 smithi114 ceph-mon[58780]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:04:43.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:42 smithi097 ceph-mon[64853]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-12T22:04:43.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:42 smithi097 ceph-mon[67407]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-12T22:04:43.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:42 smithi114 ceph-mon[58780]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-12T22:04:45.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:44 smithi097 ceph-mon[64853]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:04:45.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:44 smithi097 ceph-mon[67407]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:04:45.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:44 smithi114 ceph-mon[58780]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:04:46.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:45 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:46.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:45 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:46.247 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:45 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:47.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:46 smithi097 ceph-mon[64853]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 625 B/s rd, 0 op/s 2024-04-12T22:04:47.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:46 smithi097 ceph-mon[67407]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 625 B/s rd, 0 op/s 2024-04-12T22:04:47.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:46 smithi114 ceph-mon[58780]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 625 B/s rd, 0 op/s 2024-04-12T22:04:48.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:48 smithi114 ceph-mon[58780]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-12T22:04:48.993 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:04:48 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:04:48.659Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"smithi097:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"172.21.15.114:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-04-12T22:04:48.993 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:04:48 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:04:48.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"smithi097:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"172.21.15.114:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-12T22:04:49.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:48 smithi097 ceph-mon[64853]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-12T22:04:49.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:48 smithi097 ceph-mon[67407]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-12T22:04:49.660 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:04:49 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:04:49] "GET /metrics HTTP/1.1" 200 38414 "" "Prometheus/2.33.4" 2024-04-12T22:04:51.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:50 smithi097 ceph-mon[64853]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:04:51.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:50 smithi097 ceph-mon[67407]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:04:51.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:50 smithi114 ceph-mon[58780]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:04:52.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:51 smithi097 ceph-mon[64853]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:04:52.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:51 smithi097 ceph-mon[67407]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:04:52.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:51 smithi114 ceph-mon[58780]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:04:52.958 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:52 smithi097 ceph-mon[64853]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:04:52.958 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:52 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-12T22:04:52.958 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:52 smithi097 ceph-mon[67407]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:04:52.958 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:52 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-12T22:04:53.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:52 smithi114 ceph-mon[58780]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:04:53.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:52 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-12T22:04:54.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:53 smithi097 ceph-mon[67407]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-12T22:04:54.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:53 smithi097 ceph-mon[67407]: Upgrade: osd.2 is safe to restart 2024-04-12T22:04:54.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:53 smithi097 ceph-mon[64853]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-12T22:04:54.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:53 smithi097 ceph-mon[64853]: Upgrade: osd.2 is safe to restart 2024-04-12T22:04:54.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:53 smithi114 ceph-mon[58780]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-12T22:04:54.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:53 smithi114 ceph-mon[58780]: Upgrade: osd.2 is safe to restart 2024-04-12T22:04:55.145 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:54 smithi114 ceph-mon[58780]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:04:55.145 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:54 smithi114 ceph-mon[58780]: Upgrade: Updating osd.2 2024-04-12T22:04:55.145 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:54 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:55.145 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:54 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-12T22:04:55.145 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:54 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:04:55.145 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:54 smithi114 ceph-mon[58780]: Deploying daemon osd.2 on smithi097 2024-04-12T22:04:55.145 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:54 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:04:55.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:54 smithi097 ceph-mon[64853]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:04:55.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:54 smithi097 ceph-mon[64853]: Upgrade: Updating osd.2 2024-04-12T22:04:55.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:54 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:55.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:54 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-12T22:04:55.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:54 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:04:55.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:54 smithi097 ceph-mon[64853]: Deploying daemon osd.2 on smithi097 2024-04-12T22:04:55.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:54 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:04:55.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:54 smithi097 ceph-mon[67407]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:04:55.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:54 smithi097 ceph-mon[67407]: Upgrade: Updating osd.2 2024-04-12T22:04:55.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:54 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:04:55.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:54 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-12T22:04:55.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:54 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:04:55.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:54 smithi097 ceph-mon[67407]: Deploying daemon osd.2 on smithi097 2024-04-12T22:04:55.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:54 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:04:57.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:56 smithi097 ceph-mon[64853]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:04:57.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:56 smithi097 ceph-mon[67407]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:04:57.161 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:04:56 smithi097 systemd[1]: Stopping Ceph osd.2 for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:04:57.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:56 smithi114 ceph-mon[58780]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:04:57.660 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:04:57 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-2[38344]: 2024-04-12T22:04:57.365+0000 7f8400576700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-12T22:04:57.660 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:04:57 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-2[38344]: 2024-04-12T22:04:57.365+0000 7f8400576700 -1 osd.2 116 *** Got signal Terminated *** 2024-04-12T22:04:57.660 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:04:57 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-2[38344]: 2024-04-12T22:04:57.365+0000 7f8400576700 -1 osd.2 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-12T22:04:58.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:57 smithi097 ceph-mon[64853]: osd.2 marked itself down and dead 2024-04-12T22:04:58.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:57 smithi097 ceph-mon[67407]: osd.2 marked itself down and dead 2024-04-12T22:04:58.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:57 smithi114 ceph-mon[58780]: osd.2 marked itself down and dead 2024-04-12T22:04:58.616 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:04:58 smithi097 podman[75094]: 2024-04-12 22:04:58.315338915 +0000 UTC m=+1.178478585 container died 89daf8e339c39bc4814ee60c9e64cef80dda39e19b7b6a92f087a9eb36ac426b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-2, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, io.openshift.expose-services=, architecture=x86_64, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True) 2024-04-12T22:04:58.910 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:58 smithi097 ceph-mon[67407]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:04:58.910 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:58 smithi097 ceph-mon[67407]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-12T22:04:58.910 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:58 smithi097 ceph-mon[67407]: osdmap e117: 8 total, 7 up, 8 in 2024-04-12T22:04:58.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:58 smithi097 ceph-mon[64853]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:04:58.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:58 smithi097 ceph-mon[64853]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-12T22:04:58.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:58 smithi097 ceph-mon[64853]: osdmap e117: 8 total, 7 up, 8 in 2024-04-12T22:04:58.911 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:04:58 smithi097 podman[75094]: 2024-04-12 22:04:58.707033038 +0000 UTC m=+1.570172708 container cleanup 89daf8e339c39bc4814ee60c9e64cef80dda39e19b7b6a92f087a9eb36ac426b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-2, release=754, architecture=x86_64, RELEASE=HEAD, name=centos-stream, version=8, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-04-12T22:04:58.911 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:04:58 smithi097 bash[75094]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-2 2024-04-12T22:04:58.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:58 smithi114 ceph-mon[58780]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:04:58.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:58 smithi114 ceph-mon[58780]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-12T22:04:58.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:58 smithi114 ceph-mon[58780]: osdmap e117: 8 total, 7 up, 8 in 2024-04-12T22:04:58.994 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:04:58 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:04:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"smithi097:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"172.21.15.114:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-12T22:04:59.317 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:04:59 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:04:59] "GET /metrics HTTP/1.1" 200 38428 "" "Prometheus/2.33.4" 2024-04-12T22:04:59.317 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:04:58 smithi097 podman[75112]: 2024-04-12 22:04:58.958803657 +0000 UTC m=+0.644945158 container remove 89daf8e339c39bc4814ee60c9e64cef80dda39e19b7b6a92f087a9eb36ac426b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-2, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-04-12T22:04:59.645 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:04:59 smithi097 podman[75203]: 2024-04-12 22:04:59.219562001 +0000 UTC m=+0.017572765 image pull 668895944f0ec64b67a5b7564961d2756f2966e5d4b9338516300dbec9984951 quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a 2024-04-12T22:04:59.645 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:04:59 smithi097 podman[75203]: 2024-04-12 22:04:59.326082407 +0000 UTC m=+0.124093175 container create 704ca50eaf7faeaa854c2babc25636b1af04e0f07d06ff31d306eac797d63056 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=squid-b277feb) 2024-04-12T22:04:59.645 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:04:59 smithi097 podman[75203]: 2024-04-12 22:04:59.546117859 +0000 UTC m=+0.344128633 container init 704ca50eaf7faeaa854c2babc25636b1af04e0f07d06ff31d306eac797d63056 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, RELEASE=squid-b277feb, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True) 2024-04-12T22:04:59.646 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:04:59 smithi097 podman[75203]: 2024-04-12 22:04:59.549755588 +0000 UTC m=+0.347766362 container start 704ca50eaf7faeaa854c2babc25636b1af04e0f07d06ff31d306eac797d63056 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-b277feb, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-12T22:04:59.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:59 smithi097 ceph-mon[64853]: osdmap e118: 8 total, 7 up, 8 in 2024-04-12T22:04:59.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:04:59 smithi097 ceph-mon[64853]: pgmap v91: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-12T22:04:59.910 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:59 smithi097 ceph-mon[67407]: osdmap e118: 8 total, 7 up, 8 in 2024-04-12T22:04:59.910 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:04:59 smithi097 ceph-mon[67407]: pgmap v91: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-12T22:04:59.911 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:04:59 smithi097 podman[75203]: 2024-04-12 22:04:59.645285479 +0000 UTC m=+0.443296247 container attach 704ca50eaf7faeaa854c2babc25636b1af04e0f07d06ff31d306eac797d63056 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-2-deactivate, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-b277feb, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-12T22:04:59.911 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:04:59 smithi097 podman[75203]: 2024-04-12 22:04:59.86010313 +0000 UTC m=+0.658113896 container died 704ca50eaf7faeaa854c2babc25636b1af04e0f07d06ff31d306eac797d63056 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-2-deactivate, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-b277feb, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-12T22:05:00.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:59 smithi114 ceph-mon[58780]: osdmap e118: 8 total, 7 up, 8 in 2024-04-12T22:05:00.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:04:59 smithi114 ceph-mon[58780]: pgmap v91: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-12T22:05:00.834 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:05:00 smithi097 podman[75229]: 2024-04-12 22:05:00.482938133 +0000 UTC m=+0.617694421 container remove 704ca50eaf7faeaa854c2babc25636b1af04e0f07d06ff31d306eac797d63056 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-b277feb, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-12T22:05:00.835 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:05:00 smithi097 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@osd.2.service: Deactivated successfully. 2024-04-12T22:05:00.835 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:05:00 smithi097 systemd[1]: Stopped Ceph osd.2 for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:05:00.835 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:05:00 smithi097 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@osd.2.service: Consumed 6.364s CPU time. 2024-04-12T22:05:00.835 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:05:00 smithi097 systemd[1]: Starting Ceph osd.2 for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:05:01.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:00 smithi097 ceph-mon[64853]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:05:01.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:00 smithi097 ceph-mon[67407]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:05:01.160 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:05:00 smithi097 podman[75331]: 2024-04-12 22:05:00.834328656 +0000 UTC m=+0.017480276 image pull 668895944f0ec64b67a5b7564961d2756f2966e5d4b9338516300dbec9984951 quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a 2024-04-12T22:05:01.161 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:05:00 smithi097 podman[75331]: 2024-04-12 22:05:00.951072972 +0000 UTC m=+0.134224575 container create fc40e1109bc26bbbff7fe681427a89bb8c06a73875b58c8011e16357bdf91354 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-2-activate, GIT_BRANCH=HEAD, RELEASE=squid-b277feb, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240409) 2024-04-12T22:05:01.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:00 smithi114 ceph-mon[58780]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:05:01.502 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:05:01 smithi097 podman[75331]: 2024-04-12 22:05:01.171611359 +0000 UTC m=+0.354762964 container init fc40e1109bc26bbbff7fe681427a89bb8c06a73875b58c8011e16357bdf91354 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-2-activate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-b277feb, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-12T22:05:01.502 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:05:01 smithi097 podman[75331]: 2024-04-12 22:05:01.17499513 +0000 UTC m=+0.358146740 container start fc40e1109bc26bbbff7fe681427a89bb8c06a73875b58c8011e16357bdf91354 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-2-activate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, RELEASE=squid-b277feb, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-12T22:05:01.502 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:05:01 smithi097 podman[75331]: 2024-04-12 22:05:01.253633997 +0000 UTC m=+0.436785625 container attach fc40e1109bc26bbbff7fe681427a89bb8c06a73875b58c8011e16357bdf91354 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-2-activate, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-b277feb, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux ) 2024-04-12T22:05:01.503 INFO:teuthology.orchestra.run.smithi097.stdout:true 2024-04-12T22:05:01.865 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:05:01 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-2-activate[75345]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-12T22:05:01.865 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:05:01 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-2-activate[75345]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-12T22:05:01.865 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:05:01 smithi097 bash[75331]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-12T22:05:01.865 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:05:01 smithi097 bash[75331]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-12T22:05:01.865 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:05:01 smithi097 bash[75331]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-12T22:05:01.866 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:05:01 smithi097 bash[75331]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-12T22:05:01.866 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:05:01 smithi097 bash[75331]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-04-12T22:05:01.866 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:05:01 smithi097 bash[75331]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-12T22:05:01.866 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:05:01 smithi097 bash[75331]: --> ceph-volume raw activate successful for osd ID: 2 2024-04-12T22:05:01.866 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:05:01 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-2-activate[75345]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-12T22:05:01.866 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:05:01 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-2-activate[75345]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-12T22:05:01.866 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:05:01 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-2-activate[75345]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-04-12T22:05:01.866 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:05:01 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-2-activate[75345]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-12T22:05:01.866 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:05:01 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-2-activate[75345]: --> ceph-volume raw activate successful for osd ID: 2 2024-04-12T22:05:01.866 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:05:01 smithi097 podman[75331]: 2024-04-12 22:05:01.658384708 +0000 UTC m=+0.841536316 container died fc40e1109bc26bbbff7fe681427a89bb8c06a73875b58c8011e16357bdf91354 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-2-activate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, RELEASE=squid-b277feb, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-12T22:05:02.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:01 smithi097 ceph-mon[64853]: pgmap v92: 161 pgs: 11 active+undersized+degraded, 3 stale+active+clean, 28 active+undersized, 119 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 36/630 objects degraded (5.714%) 2024-04-12T22:05:02.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:01 smithi097 ceph-mon[64853]: from='client.44218 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:05:02.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:01 smithi097 ceph-mon[64853]: from='client.44224 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:05:02.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:01 smithi097 ceph-mon[67407]: pgmap v92: 161 pgs: 11 active+undersized+degraded, 3 stale+active+clean, 28 active+undersized, 119 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 36/630 objects degraded (5.714%) 2024-04-12T22:05:02.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:01 smithi097 ceph-mon[67407]: from='client.44218 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:05:02.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:01 smithi097 ceph-mon[67407]: from='client.44224 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:05:02.179 INFO:teuthology.orchestra.run.smithi097.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T22:05:02.179 INFO:teuthology.orchestra.run.smithi097.stdout:alertmanager.a smithi097 *:9093,9094 running (3m) 29s ago 9m 25.5M - 0.23.0 ba2b418f427c 7b4c491fe0a0 2024-04-12T22:05:02.179 INFO:teuthology.orchestra.run.smithi097.stdout:grafana.a smithi114 *:3000 unknown 99s ago 8m - - 2024-04-12T22:05:02.179 INFO:teuthology.orchestra.run.smithi097.stdout:iscsi.foo.smithi097.xndkxy smithi097 running (2m) 29s ago 7m 50.2M - 3.5 e1d6a67b021e 0436463f28cd 2024-04-12T22:05:02.179 INFO:teuthology.orchestra.run.smithi097.stdout:mgr.x smithi114 *:8443,9283,8765 running (2m) 99s ago 13m 450M - 19.0.0-2053-gb277feb2 668895944f0e b8f8df3bf928 2024-04-12T22:05:02.179 INFO:teuthology.orchestra.run.smithi097.stdout:mgr.y smithi097 *:8443,9283,8765 running (3m) 29s ago 15m 503M - 19.0.0-2053-gb277feb2 668895944f0e 488170dac11f 2024-04-12T22:05:02.179 INFO:teuthology.orchestra.run.smithi097.stdout:mon.a smithi097 running (2m) 29s ago 15m 49.4M 2048M 19.0.0-2053-gb277feb2 668895944f0e 3cfa16d3a85c 2024-04-12T22:05:02.179 INFO:teuthology.orchestra.run.smithi097.stdout:mon.b smithi114 running (110s) 99s ago 13m 18.4M 2048M 19.0.0-2053-gb277feb2 668895944f0e 71a2eff159a3 2024-04-12T22:05:02.179 INFO:teuthology.orchestra.run.smithi097.stdout:mon.c smithi097 running (2m) 29s ago 13m 38.5M 2048M 19.0.0-2053-gb277feb2 668895944f0e 69507ba88603 2024-04-12T22:05:02.179 INFO:teuthology.orchestra.run.smithi097.stdout:node-exporter.a smithi097 *:9100 running (4m) 29s ago 9m 17.9M - 1.3.1 1dbe0e931976 5edff9053b37 2024-04-12T22:05:02.179 INFO:teuthology.orchestra.run.smithi097.stdout:node-exporter.b smithi114 *:9100 running (3m) 99s ago 9m 17.0M - 1.3.1 1dbe0e931976 904afc45dc2f 2024-04-12T22:05:02.180 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0 smithi097 running (55s) 29s ago 13m 50.0M 2736M 19.0.0-2053-gb277feb2 668895944f0e 9d05fea4bf37 2024-04-12T22:05:02.180 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1 smithi097 running (34s) 29s ago 12m 12.3M 2736M 19.0.0-2053-gb277feb2 668895944f0e a9a527642889 2024-04-12T22:05:02.180 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2 smithi097 running (12m) 29s ago 12m 53.9M 2736M 17.2.0 e1d6a67b021e 89daf8e339c3 2024-04-12T22:05:02.180 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3 smithi097 running (11m) 29s ago 11m 53.5M 2736M 17.2.0 e1d6a67b021e 4e9a0b3862d9 2024-04-12T22:05:02.180 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4 smithi114 running (11m) 99s ago 11m 53.6M 2992M 17.2.0 e1d6a67b021e fb42c0e8e575 2024-04-12T22:05:02.180 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5 smithi114 running (11m) 99s ago 11m 50.2M 2992M 17.2.0 e1d6a67b021e 3acd4a3b61e5 2024-04-12T22:05:02.180 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6 smithi114 running (10m) 99s ago 10m 49.6M 2992M 17.2.0 e1d6a67b021e 5969baf67f41 2024-04-12T22:05:02.180 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7 smithi114 running (10m) 99s ago 10m 51.8M 2992M 17.2.0 e1d6a67b021e f68340fecf2c 2024-04-12T22:05:02.180 INFO:teuthology.orchestra.run.smithi097.stdout:prometheus.a smithi114 *:9095 running (2m) 99s ago 9m 65.1M - 2.33.4 514e6a882f6e 0ad5437ed63f 2024-04-12T22:05:02.180 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.foo.smithi097.sggtkh smithi097 *:8000 running (8m) 29s ago 8m 72.7M - 17.2.0 e1d6a67b021e a39f9b2d8509 2024-04-12T22:05:02.180 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.foo.smithi114.ulfbyo smithi114 *:8000 running (8m) 99s ago 8m 72.8M - 17.2.0 e1d6a67b021e df79a48bdd59 2024-04-12T22:05:02.180 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.smpl.smithi097.exxkgl smithi097 *:80 running (7m) 29s ago 7m 72.7M - 17.2.0 e1d6a67b021e a036c58a0099 2024-04-12T22:05:02.180 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.smpl.smithi114.lrgvph smithi114 *:80 running (7m) 99s ago 7m 74.4M - 17.2.0 e1d6a67b021e 031a5fc24237 2024-04-12T22:05:02.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:01 smithi114 ceph-mon[58780]: pgmap v92: 161 pgs: 11 active+undersized+degraded, 3 stale+active+clean, 28 active+undersized, 119 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 36/630 objects degraded (5.714%) 2024-04-12T22:05:02.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:01 smithi114 ceph-mon[58780]: from='client.44218 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:05:02.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:01 smithi114 ceph-mon[58780]: from='client.44224 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:05:02.597 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:05:02 smithi097 podman[75531]: 2024-04-12 22:05:02.399798117 +0000 UTC m=+0.736219157 container remove fc40e1109bc26bbbff7fe681427a89bb8c06a73875b58c8011e16357bdf91354 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-2-activate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, RELEASE=squid-b277feb, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-12T22:05:02.597 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-12T22:05:02.597 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": { 2024-04-12T22:05:02.598 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 3 2024-04-12T22:05:02.598 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:05:02.598 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": { 2024-04-12T22:05:02.598 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 2 2024-04-12T22:05:02.598 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:05:02.598 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": { 2024-04-12T22:05:02.598 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-04-12T22:05:02.598 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 2 2024-04-12T22:05:02.598 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:05:02.598 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw": { 2024-04-12T22:05:02.598 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-12T22:05:02.598 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:05:02.599 INFO:teuthology.orchestra.run.smithi097.stdout: "overall": { 2024-04-12T22:05:02.599 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 9, 2024-04-12T22:05:02.599 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 7 2024-04-12T22:05:02.599 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-12T22:05:02.599 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-12T22:05:02.866 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:05:02 smithi097 podman[75641]: 2024-04-12 22:05:02.520267755 +0000 UTC m=+0.017821092 image pull 668895944f0ec64b67a5b7564961d2756f2966e5d4b9338516300dbec9984951 quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a 2024-04-12T22:05:02.866 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:05:02 smithi097 podman[75641]: 2024-04-12 22:05:02.62566141 +0000 UTC m=+0.123214734 container create 2e2297afff5034813210f76ca45cd09c69b11925772c6e66624fe3c01c77bcd3 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=squid-b277feb, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-12T22:05:02.939 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-12T22:05:02.939 INFO:teuthology.orchestra.run.smithi097.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a", 2024-04-12T22:05:02.939 INFO:teuthology.orchestra.run.smithi097.stdout: "in_progress": true, 2024-04-12T22:05:02.939 INFO:teuthology.orchestra.run.smithi097.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T22:05:02.939 INFO:teuthology.orchestra.run.smithi097.stdout: "services_complete": [ 2024-04-12T22:05:02.939 INFO:teuthology.orchestra.run.smithi097.stdout: "mon", 2024-04-12T22:05:02.939 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr" 2024-04-12T22:05:02.939 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-04-12T22:05:02.939 INFO:teuthology.orchestra.run.smithi097.stdout: "progress": "7/23 daemons upgraded", 2024-04-12T22:05:02.940 INFO:teuthology.orchestra.run.smithi097.stdout: "message": "Currently upgrading osd daemons", 2024-04-12T22:05:02.940 INFO:teuthology.orchestra.run.smithi097.stdout: "is_paused": false 2024-04-12T22:05:02.940 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-12T22:05:03.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:02 smithi097 ceph-mon[64853]: Health check failed: Degraded data redundancy: 36/630 objects degraded (5.714%), 11 pgs degraded (PG_DEGRADED) 2024-04-12T22:05:03.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:02 smithi097 ceph-mon[64853]: from='client.44230 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:05:03.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:02 smithi097 ceph-mon[64853]: from='client.? 172.21.15.97:0/1167372640' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:05:03.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:02 smithi097 ceph-mon[67407]: Health check failed: Degraded data redundancy: 36/630 objects degraded (5.714%), 11 pgs degraded (PG_DEGRADED) 2024-04-12T22:05:03.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:02 smithi097 ceph-mon[67407]: from='client.44230 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:05:03.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:02 smithi097 ceph-mon[67407]: from='client.? 172.21.15.97:0/1167372640' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:05:03.161 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:05:02 smithi097 podman[75641]: 2024-04-12 22:05:02.88845878 +0000 UTC m=+0.386012118 container init 2e2297afff5034813210f76ca45cd09c69b11925772c6e66624fe3c01c77bcd3 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-b277feb, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-12T22:05:03.161 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:05:02 smithi097 podman[75641]: 2024-04-12 22:05:02.892406405 +0000 UTC m=+0.389959733 container start 2e2297afff5034813210f76ca45cd09c69b11925772c6e66624fe3c01c77bcd3 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=squid-b277feb, GIT_CLEAN=True) 2024-04-12T22:05:03.161 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:05:02 smithi097 ceph-osd[75683]: -- 172.21.15.97:0/3722917452 <== mon.0 v2:172.21.15.97:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b297a29680 con 0x55b2979f3000 2024-04-12T22:05:03.161 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:05:03 smithi097 bash[75641]: 2e2297afff5034813210f76ca45cd09c69b11925772c6e66624fe3c01c77bcd3 2024-04-12T22:05:03.161 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:05:03 smithi097 systemd[1]: Started Ceph osd.2 for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:05:03.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:02 smithi114 ceph-mon[58780]: Health check failed: Degraded data redundancy: 36/630 objects degraded (5.714%), 11 pgs degraded (PG_DEGRADED) 2024-04-12T22:05:03.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:02 smithi114 ceph-mon[58780]: from='client.44230 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:05:03.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:02 smithi114 ceph-mon[58780]: from='client.? 172.21.15.97:0/1167372640' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:05:03.348 INFO:teuthology.orchestra.run.smithi097.stdout:HEALTH_WARN 1 failed cephadm daemon(s); 1 osds down; Degraded data redundancy: 36/630 objects degraded (5.714%), 11 pgs degraded 2024-04-12T22:05:03.348 INFO:teuthology.orchestra.run.smithi097.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-12T22:05:03.348 INFO:teuthology.orchestra.run.smithi097.stdout: daemon grafana.a on smithi114 is in unknown state 2024-04-12T22:05:03.348 INFO:teuthology.orchestra.run.smithi097.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-12T22:05:03.348 INFO:teuthology.orchestra.run.smithi097.stdout: osd.2 (root=default,host=smithi097) is down 2024-04-12T22:05:03.348 INFO:teuthology.orchestra.run.smithi097.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 36/630 objects degraded (5.714%), 11 pgs degraded 2024-04-12T22:05:03.348 INFO:teuthology.orchestra.run.smithi097.stdout: pg 2.3 is active+undersized+degraded, acting [5,7] 2024-04-12T22:05:03.348 INFO:teuthology.orchestra.run.smithi097.stdout: pg 2.e is active+undersized+degraded, acting [3,7] 2024-04-12T22:05:03.349 INFO:teuthology.orchestra.run.smithi097.stdout: pg 2.1c is active+undersized+degraded, acting [7,5] 2024-04-12T22:05:03.349 INFO:teuthology.orchestra.run.smithi097.stdout: pg 3.5 is active+undersized+degraded, acting [5,3] 2024-04-12T22:05:03.349 INFO:teuthology.orchestra.run.smithi097.stdout: pg 3.9 is active+undersized+degraded, acting [4,7] 2024-04-12T22:05:03.349 INFO:teuthology.orchestra.run.smithi097.stdout: pg 3.13 is active+undersized+degraded, acting [7,4] 2024-04-12T22:05:03.349 INFO:teuthology.orchestra.run.smithi097.stdout: pg 3.1a is active+undersized+degraded, acting [4,1] 2024-04-12T22:05:03.349 INFO:teuthology.orchestra.run.smithi097.stdout: pg 3.1e is active+undersized+degraded, acting [3,6] 2024-04-12T22:05:03.349 INFO:teuthology.orchestra.run.smithi097.stdout: pg 3.1f is active+undersized+degraded, acting [0,5] 2024-04-12T22:05:03.349 INFO:teuthology.orchestra.run.smithi097.stdout: pg 4.d is active+undersized+degraded, acting [4,1] 2024-04-12T22:05:03.349 INFO:teuthology.orchestra.run.smithi097.stdout: pg 5.1c is active+undersized+degraded, acting [4,3] 2024-04-12T22:05:03.660 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:05:03 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-2[75679]: 2024-04-12T22:05:03.445+0000 7f434dcdc7c0 -1 Falling back to public interface 2024-04-12T22:05:04.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:04 smithi097 ceph-mon[64853]: from='client.34314 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:05:04.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:04 smithi097 ceph-mon[64853]: pgmap v93: 161 pgs: 14 active+undersized+degraded, 33 active+undersized, 114 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 56/630 objects degraded (8.889%) 2024-04-12T22:05:04.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:04 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:04.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:04 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:04.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:04 smithi097 ceph-mon[64853]: from='client.? 172.21.15.97:0/3739622100' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T22:05:04.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:04 smithi097 ceph-mon[67407]: from='client.34314 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:05:04.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:04 smithi097 ceph-mon[67407]: pgmap v93: 161 pgs: 14 active+undersized+degraded, 33 active+undersized, 114 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 56/630 objects degraded (8.889%) 2024-04-12T22:05:04.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:04 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:04.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:04 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:04.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:04 smithi097 ceph-mon[67407]: from='client.? 172.21.15.97:0/3739622100' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T22:05:04.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:04 smithi114 ceph-mon[58780]: from='client.34314 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:05:04.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:04 smithi114 ceph-mon[58780]: pgmap v93: 161 pgs: 14 active+undersized+degraded, 33 active+undersized, 114 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 56/630 objects degraded (8.889%) 2024-04-12T22:05:04.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:04 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:04.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:04 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:04.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:04 smithi114 ceph-mon[58780]: from='client.? 172.21.15.97:0/3739622100' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T22:05:06.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:06 smithi097 ceph-mon[64853]: pgmap v94: 161 pgs: 14 active+undersized+degraded, 33 active+undersized, 114 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 56/630 objects degraded (8.889%) 2024-04-12T22:05:06.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:06 smithi097 ceph-mon[67407]: pgmap v94: 161 pgs: 14 active+undersized+degraded, 33 active+undersized, 114 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 56/630 objects degraded (8.889%) 2024-04-12T22:05:06.411 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:05:06 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-2[75679]: 2024-04-12T22:05:06.152+0000 7f434dcdc7c0 -1 osd.2 0 read_superblock omap replica is missing. 2024-04-12T22:05:06.411 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:05:06 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-2[75679]: 2024-04-12T22:05:06.172+0000 7f434dcdc7c0 -1 osd.2 116 log_to_monitors true 2024-04-12T22:05:06.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:06 smithi114 ceph-mon[58780]: pgmap v94: 161 pgs: 14 active+undersized+degraded, 33 active+undersized, 114 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 56/630 objects degraded (8.889%) 2024-04-12T22:05:07.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:07 smithi097 ceph-mon[64853]: from='osd.2 [v2:172.21.15.97:6818/1196483794,v1:172.21.15.97:6819/1196483794]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-12T22:05:07.410 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:07 smithi097 ceph-mon[67407]: from='osd.2 [v2:172.21.15.97:6818/1196483794,v1:172.21.15.97:6819/1196483794]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-12T22:05:07.411 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:05:07 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-2[75679]: 2024-04-12T22:05:07.164+0000 7f433f134700 -1 osd.2 116 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T22:05:07.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:07 smithi114 ceph-mon[58780]: from='osd.2 [v2:172.21.15.97:6818/1196483794,v1:172.21.15.97:6819/1196483794]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-12T22:05:08.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:08 smithi097 ceph-mon[64853]: pgmap v95: 161 pgs: 14 active+undersized+degraded, 33 active+undersized, 114 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 56/630 objects degraded (8.889%) 2024-04-12T22:05:08.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:08 smithi097 ceph-mon[64853]: from='osd.2 [v2:172.21.15.97:6818/1196483794,v1:172.21.15.97:6819/1196483794]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-12T22:05:08.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:08 smithi097 ceph-mon[64853]: osdmap e119: 8 total, 7 up, 8 in 2024-04-12T22:05:08.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:08 smithi097 ceph-mon[64853]: from='osd.2 [v2:172.21.15.97:6818/1196483794,v1:172.21.15.97:6819/1196483794]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-12T22:05:08.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:08 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:08.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:08 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:08.410 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:08 smithi097 ceph-mon[67407]: pgmap v95: 161 pgs: 14 active+undersized+degraded, 33 active+undersized, 114 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 56/630 objects degraded (8.889%) 2024-04-12T22:05:08.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:08 smithi097 ceph-mon[67407]: from='osd.2 [v2:172.21.15.97:6818/1196483794,v1:172.21.15.97:6819/1196483794]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-12T22:05:08.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:08 smithi097 ceph-mon[67407]: osdmap e119: 8 total, 7 up, 8 in 2024-04-12T22:05:08.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:08 smithi097 ceph-mon[67407]: from='osd.2 [v2:172.21.15.97:6818/1196483794,v1:172.21.15.97:6819/1196483794]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-12T22:05:08.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:08 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:08.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:08 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:08.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:08 smithi114 ceph-mon[58780]: pgmap v95: 161 pgs: 14 active+undersized+degraded, 33 active+undersized, 114 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 56/630 objects degraded (8.889%) 2024-04-12T22:05:08.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:08 smithi114 ceph-mon[58780]: from='osd.2 [v2:172.21.15.97:6818/1196483794,v1:172.21.15.97:6819/1196483794]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-12T22:05:08.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:08 smithi114 ceph-mon[58780]: osdmap e119: 8 total, 7 up, 8 in 2024-04-12T22:05:08.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:08 smithi114 ceph-mon[58780]: from='osd.2 [v2:172.21.15.97:6818/1196483794,v1:172.21.15.97:6819/1196483794]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-12T22:05:08.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:08 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:08.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:08 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:08.993 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:05:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:05:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"smithi097:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"172.21.15.114:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-12T22:05:09.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:09 smithi097 ceph-mon[64853]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-12T22:05:09.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:09 smithi097 ceph-mon[64853]: osd.2 [v2:172.21.15.97:6818/1196483794,v1:172.21.15.97:6819/1196483794] boot 2024-04-12T22:05:09.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:09 smithi097 ceph-mon[64853]: osdmap e120: 8 total, 8 up, 8 in 2024-04-12T22:05:09.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:09 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T22:05:09.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:09 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:09.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:09 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:09.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:09 smithi097 ceph-mon[64853]: osdmap e121: 8 total, 8 up, 8 in 2024-04-12T22:05:09.411 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:05:09 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:05:09] "GET /metrics HTTP/1.1" 200 38428 "" "Prometheus/2.33.4" 2024-04-12T22:05:09.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:09 smithi097 ceph-mon[67407]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-12T22:05:09.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:09 smithi097 ceph-mon[67407]: osd.2 [v2:172.21.15.97:6818/1196483794,v1:172.21.15.97:6819/1196483794] boot 2024-04-12T22:05:09.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:09 smithi097 ceph-mon[67407]: osdmap e120: 8 total, 8 up, 8 in 2024-04-12T22:05:09.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:09 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T22:05:09.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:09 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:09.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:09 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:09.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:09 smithi097 ceph-mon[67407]: osdmap e121: 8 total, 8 up, 8 in 2024-04-12T22:05:09.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:09 smithi114 ceph-mon[58780]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-12T22:05:09.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:09 smithi114 ceph-mon[58780]: osd.2 [v2:172.21.15.97:6818/1196483794,v1:172.21.15.97:6819/1196483794] boot 2024-04-12T22:05:09.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:09 smithi114 ceph-mon[58780]: osdmap e120: 8 total, 8 up, 8 in 2024-04-12T22:05:09.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:09 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T22:05:09.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:09 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:09.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:09 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:09.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:09 smithi114 ceph-mon[58780]: osdmap e121: 8 total, 8 up, 8 in 2024-04-12T22:05:10.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:10 smithi097 ceph-mon[64853]: pgmap v98: 161 pgs: 14 active+undersized+degraded, 33 active+undersized, 114 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 56/630 objects degraded (8.889%) 2024-04-12T22:05:10.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:10 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:10.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:10 smithi097 ceph-mon[64853]: Health check update: Degraded data redundancy: 56/630 objects degraded (8.889%), 14 pgs degraded (PG_DEGRADED) 2024-04-12T22:05:10.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:10 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:05:10.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:10 smithi097 ceph-mon[67407]: pgmap v98: 161 pgs: 14 active+undersized+degraded, 33 active+undersized, 114 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 56/630 objects degraded (8.889%) 2024-04-12T22:05:10.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:10 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:10.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:10 smithi097 ceph-mon[67407]: Health check update: Degraded data redundancy: 56/630 objects degraded (8.889%), 14 pgs degraded (PG_DEGRADED) 2024-04-12T22:05:10.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:10 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:05:11.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:10 smithi114 ceph-mon[58780]: pgmap v98: 161 pgs: 14 active+undersized+degraded, 33 active+undersized, 114 active+clean; 585 KiB data, 151 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 56/630 objects degraded (8.889%) 2024-04-12T22:05:11.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:10 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:11.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:10 smithi114 ceph-mon[58780]: Health check update: Degraded data redundancy: 56/630 objects degraded (8.889%), 14 pgs degraded (PG_DEGRADED) 2024-04-12T22:05:11.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:10 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:05:11.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:11 smithi097 ceph-mon[64853]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:05:11.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:11 smithi097 ceph-mon[67407]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:05:12.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:11 smithi114 ceph-mon[58780]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:05:12.896 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:12 smithi097 ceph-mon[67407]: pgmap v100: 161 pgs: 4 active+undersized+degraded, 14 active+undersized, 143 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 21/630 objects degraded (3.333%) 2024-04-12T22:05:12.896 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:12 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:12.896 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:12 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:12.896 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:12 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:05:12.896 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:12 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:05:12.896 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:12 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:12.896 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:12 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:05:12.896 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:12 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:05:12.897 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:12 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:05:12.897 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:12 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:05:12.897 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:12 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-12T22:05:12.897 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:12 smithi097 ceph-mon[64853]: pgmap v100: 161 pgs: 4 active+undersized+degraded, 14 active+undersized, 143 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 21/630 objects degraded (3.333%) 2024-04-12T22:05:12.897 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:12 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:12.897 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:12 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:12.897 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:12 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:05:12.897 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:12 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:05:12.897 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:12 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:12.897 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:12 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:05:12.898 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:12 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:05:12.898 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:12 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:05:12.898 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:12 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:05:12.898 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:12 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-12T22:05:13.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:12 smithi114 ceph-mon[58780]: pgmap v100: 161 pgs: 4 active+undersized+degraded, 14 active+undersized, 143 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 21/630 objects degraded (3.333%) 2024-04-12T22:05:13.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:12 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:13.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:12 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:13.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:12 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:05:13.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:12 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:05:13.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:12 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:13.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:12 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:05:13.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:12 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:05:13.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:12 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:05:13.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:12 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:05:13.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:12 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-12T22:05:14.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:13 smithi097 ceph-mon[64853]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 977 B/s rd, 0 op/s 2024-04-12T22:05:14.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:13 smithi097 ceph-mon[64853]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1012 B/s rd, 0 op/s 2024-04-12T22:05:14.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:13 smithi097 ceph-mon[64853]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-12T22:05:14.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:13 smithi097 ceph-mon[64853]: Upgrade: osd.3 is safe to restart 2024-04-12T22:05:14.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:13 smithi097 ceph-mon[64853]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 21/630 objects degraded (3.333%), 4 pgs degraded) 2024-04-12T22:05:14.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:13 smithi097 ceph-mon[67407]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 977 B/s rd, 0 op/s 2024-04-12T22:05:14.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:13 smithi097 ceph-mon[67407]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1012 B/s rd, 0 op/s 2024-04-12T22:05:14.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:13 smithi097 ceph-mon[67407]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-12T22:05:14.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:13 smithi097 ceph-mon[67407]: Upgrade: osd.3 is safe to restart 2024-04-12T22:05:14.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:13 smithi097 ceph-mon[67407]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 21/630 objects degraded (3.333%), 4 pgs degraded) 2024-04-12T22:05:14.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:13 smithi114 ceph-mon[58780]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 977 B/s rd, 0 op/s 2024-04-12T22:05:14.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:13 smithi114 ceph-mon[58780]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1012 B/s rd, 0 op/s 2024-04-12T22:05:14.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:13 smithi114 ceph-mon[58780]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-12T22:05:14.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:13 smithi114 ceph-mon[58780]: Upgrade: osd.3 is safe to restart 2024-04-12T22:05:14.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:13 smithi114 ceph-mon[58780]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 21/630 objects degraded (3.333%), 4 pgs degraded) 2024-04-12T22:05:15.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:14 smithi097 ceph-mon[67407]: Upgrade: Updating osd.3 2024-04-12T22:05:15.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:14 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:15.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:14 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-12T22:05:15.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:14 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:05:15.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:14 smithi097 ceph-mon[67407]: Deploying daemon osd.3 on smithi097 2024-04-12T22:05:15.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:14 smithi097 ceph-mon[67407]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 845 B/s rd, 0 op/s 2024-04-12T22:05:15.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:14 smithi097 ceph-mon[64853]: Upgrade: Updating osd.3 2024-04-12T22:05:15.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:14 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:15.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:14 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-12T22:05:15.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:14 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:05:15.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:14 smithi097 ceph-mon[64853]: Deploying daemon osd.3 on smithi097 2024-04-12T22:05:15.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:14 smithi097 ceph-mon[64853]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 845 B/s rd, 0 op/s 2024-04-12T22:05:15.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:14 smithi114 ceph-mon[58780]: Upgrade: Updating osd.3 2024-04-12T22:05:15.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:14 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:15.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:14 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-12T22:05:15.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:14 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:05:15.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:14 smithi114 ceph-mon[58780]: Deploying daemon osd.3 on smithi097 2024-04-12T22:05:15.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:14 smithi114 ceph-mon[58780]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 845 B/s rd, 0 op/s 2024-04-12T22:05:17.279 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:17 smithi097 ceph-mon[64853]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-12T22:05:17.279 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:17 smithi097 ceph-mon[67407]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-12T22:05:17.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:17 smithi114 ceph-mon[58780]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-12T22:05:17.660 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:05:17 smithi097 systemd[1]: Stopping Ceph osd.3 for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:05:18.160 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:05:17 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-3[41341]: 2024-04-12T22:05:17.717+0000 7f234cf76700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-12T22:05:18.160 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:05:17 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-3[41341]: 2024-04-12T22:05:17.717+0000 7f234cf76700 -1 osd.3 121 *** Got signal Terminated *** 2024-04-12T22:05:18.160 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:05:17 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-3[41341]: 2024-04-12T22:05:17.717+0000 7f234cf76700 -1 osd.3 121 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-12T22:05:18.596 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:18 smithi097 ceph-mon[64853]: osd.3 marked itself down and dead 2024-04-12T22:05:18.596 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:18 smithi097 ceph-mon[67407]: osd.3 marked itself down and dead 2024-04-12T22:05:18.661 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:18 smithi114 ceph-mon[58780]: osd.3 marked itself down and dead 2024-04-12T22:05:18.894 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:05:18 smithi097 podman[77558]: 2024-04-12 22:05:18.595847494 +0000 UTC m=+0.990560149 container died 4e9a0b3862d98e5de2b8061d15937d28a37dc555cfba271d8fc8efb24fe44758 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-3, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , release=754, version=8, com.redhat.component=centos-stream-container, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, ceph=True, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=) 2024-04-12T22:05:18.993 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:05:18 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:05:18.661Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"smithi097:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"172.21.15.114:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-12T22:05:19.160 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:05:18 smithi097 podman[77558]: 2024-04-12 22:05:18.967359477 +0000 UTC m=+1.362072131 container cleanup 4e9a0b3862d98e5de2b8061d15937d28a37dc555cfba271d8fc8efb24fe44758 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-3, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, com.redhat.component=centos-stream-container, ceph=True, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, name=centos-stream, GIT_BRANCH=HEAD, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, vendor=Red Hat, Inc., GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, distribution-scope=public) 2024-04-12T22:05:19.160 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:05:18 smithi097 bash[77558]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-3 2024-04-12T22:05:19.586 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:05:19 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:05:19] "GET /metrics HTTP/1.1" 200 38444 "" "Prometheus/2.33.4" 2024-04-12T22:05:19.586 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:19 smithi097 ceph-mon[64853]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-12T22:05:19.586 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:19 smithi097 ceph-mon[64853]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-12T22:05:19.586 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:19 smithi097 ceph-mon[64853]: osdmap e122: 8 total, 7 up, 8 in 2024-04-12T22:05:19.587 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:19 smithi097 ceph-mon[67407]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-12T22:05:19.587 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:19 smithi097 ceph-mon[67407]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-12T22:05:19.587 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:19 smithi097 ceph-mon[67407]: osdmap e122: 8 total, 7 up, 8 in 2024-04-12T22:05:19.587 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:05:19 smithi097 podman[77575]: 2024-04-12 22:05:19.176814103 +0000 UTC m=+0.579618559 container remove 4e9a0b3862d98e5de2b8061d15937d28a37dc555cfba271d8fc8efb24fe44758 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-3, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, release=754, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, io.openshift.expose-services=, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-04-12T22:05:19.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:19 smithi114 ceph-mon[58780]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-12T22:05:19.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:19 smithi114 ceph-mon[58780]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-12T22:05:19.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:19 smithi114 ceph-mon[58780]: osdmap e122: 8 total, 7 up, 8 in 2024-04-12T22:05:19.910 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:05:19 smithi097 podman[77665]: 2024-04-12 22:05:19.586129426 +0000 UTC m=+0.110330227 container create 220526c89017f44a8f386d256be00305333252ed2dcca16a57eab30a4bb0c74c (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-b277feb, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5) 2024-04-12T22:05:19.910 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:05:19 smithi097 podman[77665]: 2024-04-12 22:05:19.492766003 +0000 UTC m=+0.016966819 image pull 668895944f0ec64b67a5b7564961d2756f2966e5d4b9338516300dbec9984951 quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a 2024-04-12T22:05:19.910 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:05:19 smithi097 podman[77665]: 2024-04-12 22:05:19.830621363 +0000 UTC m=+0.354822164 container init 220526c89017f44a8f386d256be00305333252ed2dcca16a57eab30a4bb0c74c (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-3-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240409, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-b277feb, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-12T22:05:19.910 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:05:19 smithi097 podman[77665]: 2024-04-12 22:05:19.833965767 +0000 UTC m=+0.358166579 container start 220526c89017f44a8f386d256be00305333252ed2dcca16a57eab30a4bb0c74c (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-3-deactivate, RELEASE=squid-b277feb, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-12T22:05:20.254 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:05:19 smithi097 podman[77665]: 2024-04-12 22:05:19.914113579 +0000 UTC m=+0.438314382 container attach 220526c89017f44a8f386d256be00305333252ed2dcca16a57eab30a4bb0c74c (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-3-deactivate, io.buildah.version=1.33.5, RELEASE=squid-b277feb, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240409, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-12T22:05:20.254 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:05:20 smithi097 podman[77665]: 2024-04-12 22:05:20.125074684 +0000 UTC m=+0.649275483 container died 220526c89017f44a8f386d256be00305333252ed2dcca16a57eab30a4bb0c74c (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-b277feb, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-12T22:05:20.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:20 smithi097 ceph-mon[64853]: osdmap e123: 8 total, 7 up, 8 in 2024-04-12T22:05:20.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:20 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:20.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:20 smithi097 ceph-mon[67407]: osdmap e123: 8 total, 7 up, 8 in 2024-04-12T22:05:20.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:20 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:20.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:20 smithi114 ceph-mon[58780]: osdmap e123: 8 total, 7 up, 8 in 2024-04-12T22:05:20.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:20 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:21.379 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:05:20 smithi097 podman[77691]: 2024-04-12 22:05:20.94317894 +0000 UTC m=+0.808757742 container remove 220526c89017f44a8f386d256be00305333252ed2dcca16a57eab30a4bb0c74c (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-3-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-b277feb, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-12T22:05:21.379 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:05:20 smithi097 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@osd.3.service: Deactivated successfully. 2024-04-12T22:05:21.379 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:05:20 smithi097 systemd[1]: Stopped Ceph osd.3 for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:05:21.379 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:05:20 smithi097 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@osd.3.service: Consumed 7.871s CPU time. 2024-04-12T22:05:21.379 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:05:21 smithi097 systemd[1]: Starting Ceph osd.3 for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:05:21.660 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:05:21 smithi097 podman[77795]: 2024-04-12 22:05:21.377852158 +0000 UTC m=+0.111436615 container create 7428f3a78915d3cccc6ee4e1b71cf59a5d541a89b83f44e26a2374795d570c9d (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-3-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, RELEASE=squid-b277feb, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux ) 2024-04-12T22:05:21.660 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:05:21 smithi097 podman[77795]: 2024-04-12 22:05:21.284363941 +0000 UTC m=+0.017948396 image pull 668895944f0ec64b67a5b7564961d2756f2966e5d4b9338516300dbec9984951 quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a 2024-04-12T22:05:21.660 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:05:21 smithi097 podman[77795]: 2024-04-12 22:05:21.605385034 +0000 UTC m=+0.338969492 container init 7428f3a78915d3cccc6ee4e1b71cf59a5d541a89b83f44e26a2374795d570c9d (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-3-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-b277feb, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-12T22:05:21.660 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:05:21 smithi097 podman[77795]: 2024-04-12 22:05:21.608539632 +0000 UTC m=+0.342124088 container start 7428f3a78915d3cccc6ee4e1b71cf59a5d541a89b83f44e26a2374795d570c9d (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-3-activate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-b277feb, ceph=True, GIT_BRANCH=HEAD) 2024-04-12T22:05:22.054 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:21 smithi097 ceph-mon[64853]: pgmap v108: 161 pgs: 27 stale+active+clean, 134 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-12T22:05:22.054 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:21 smithi097 ceph-mon[64853]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:05:22.054 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:21 smithi097 ceph-mon[67407]: pgmap v108: 161 pgs: 27 stale+active+clean, 134 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-12T22:05:22.054 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:21 smithi097 ceph-mon[67407]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:05:22.055 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:05:21 smithi097 podman[77795]: 2024-04-12 22:05:21.690961433 +0000 UTC m=+0.424545903 container attach 7428f3a78915d3cccc6ee4e1b71cf59a5d541a89b83f44e26a2374795d570c9d (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-3-activate, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, RELEASE=squid-b277feb, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-12T22:05:22.055 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:05:22 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-3-activate[77809]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-12T22:05:22.055 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:05:22 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-3-activate[77809]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-04-12T22:05:22.055 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:05:22 smithi097 bash[77795]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-12T22:05:22.056 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:05:22 smithi097 bash[77795]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-04-12T22:05:22.056 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:05:22 smithi097 bash[77795]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-04-12T22:05:22.056 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:05:22 smithi097 bash[77795]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-04-12T22:05:22.056 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:05:22 smithi097 bash[77795]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-04-12T22:05:22.056 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:05:22 smithi097 bash[77795]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-12T22:05:22.056 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:05:22 smithi097 bash[77795]: --> ceph-volume raw activate successful for osd ID: 3 2024-04-12T22:05:22.056 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:05:22 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-3-activate[77809]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-04-12T22:05:22.056 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:05:22 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-3-activate[77809]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-04-12T22:05:22.056 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:05:22 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-3-activate[77809]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-04-12T22:05:22.056 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:05:22 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-3-activate[77809]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-12T22:05:22.056 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:05:22 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-3-activate[77809]: --> ceph-volume raw activate successful for osd ID: 3 2024-04-12T22:05:22.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:21 smithi114 ceph-mon[58780]: pgmap v108: 161 pgs: 27 stale+active+clean, 134 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-12T22:05:22.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:21 smithi114 ceph-mon[58780]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:05:22.410 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:05:22 smithi097 podman[77795]: 2024-04-12 22:05:22.080527257 +0000 UTC m=+0.814111713 container died 7428f3a78915d3cccc6ee4e1b71cf59a5d541a89b83f44e26a2374795d570c9d (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-3-activate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-b277feb, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, GIT_BRANCH=HEAD) 2024-04-12T22:05:22.952 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:22 smithi097 ceph-mon[64853]: pgmap v109: 161 pgs: 26 active+undersized+degraded, 40 active+undersized, 95 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 81/630 objects degraded (12.857%) 2024-04-12T22:05:22.952 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:22 smithi097 ceph-mon[64853]: Health check failed: Degraded data redundancy: 81/630 objects degraded (12.857%), 26 pgs degraded (PG_DEGRADED) 2024-04-12T22:05:22.952 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:22 smithi097 ceph-mon[67407]: pgmap v109: 161 pgs: 26 active+undersized+degraded, 40 active+undersized, 95 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 81/630 objects degraded (12.857%) 2024-04-12T22:05:22.952 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:22 smithi097 ceph-mon[67407]: Health check failed: Degraded data redundancy: 81/630 objects degraded (12.857%), 26 pgs degraded (PG_DEGRADED) 2024-04-12T22:05:22.952 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:05:22 smithi097 podman[77930]: 2024-04-12 22:05:22.701022224 +0000 UTC m=+0.615586230 container remove 7428f3a78915d3cccc6ee4e1b71cf59a5d541a89b83f44e26a2374795d570c9d (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-3-activate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, GIT_CLEAN=True, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-b277feb, maintainer=Guillaume Abrioux ) 2024-04-12T22:05:22.953 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:05:22 smithi097 podman[77994]: 2024-04-12 22:05:22.813458602 +0000 UTC m=+0.018159659 image pull 668895944f0ec64b67a5b7564961d2756f2966e5d4b9338516300dbec9984951 quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a 2024-04-12T22:05:22.953 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:05:22 smithi097 podman[77994]: 2024-04-12 22:05:22.952037049 +0000 UTC m=+0.156738077 container create 415b4d3d85f2589a0068603a86da54acbaf07ac976cbaabf272ce9e5157201b8 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-3, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-b277feb, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-12T22:05:23.238 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:05:23 smithi097 podman[77994]: 2024-04-12 22:05:23.146306937 +0000 UTC m=+0.351007966 container init 415b4d3d85f2589a0068603a86da54acbaf07ac976cbaabf272ce9e5157201b8 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-3, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, RELEASE=squid-b277feb, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409) 2024-04-12T22:05:23.238 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:05:23 smithi097 podman[77994]: 2024-04-12 22:05:23.149298887 +0000 UTC m=+0.353999915 container start 415b4d3d85f2589a0068603a86da54acbaf07ac976cbaabf272ce9e5157201b8 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-3, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-b277feb, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-12T22:05:23.239 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:05:23 smithi097 ceph-osd[78013]: -- 172.21.15.97:0/709294342 <== mon.0 v2:172.21.15.97:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555c19593680 con 0x555c1955d400 2024-04-12T22:05:23.242 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:22 smithi114 ceph-mon[58780]: pgmap v109: 161 pgs: 26 active+undersized+degraded, 40 active+undersized, 95 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 81/630 objects degraded (12.857%) 2024-04-12T22:05:23.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:22 smithi114 ceph-mon[58780]: Health check failed: Degraded data redundancy: 81/630 objects degraded (12.857%), 26 pgs degraded (PG_DEGRADED) 2024-04-12T22:05:23.660 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:05:23 smithi097 bash[77994]: 415b4d3d85f2589a0068603a86da54acbaf07ac976cbaabf272ce9e5157201b8 2024-04-12T22:05:23.660 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:05:23 smithi097 systemd[1]: Started Ceph osd.3 for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:05:24.118 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:05:23 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-3[78009]: 2024-04-12T22:05:23.699+0000 7f94f5d9a7c0 -1 Falling back to public interface 2024-04-12T22:05:24.383 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:24 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:24.383 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:24 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:24.383 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:24 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:24.383 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:24 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:24.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:24 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:24.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:24 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:26.025 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:25 smithi097 ceph-mon[64853]: pgmap v110: 161 pgs: 26 active+undersized+degraded, 40 active+undersized, 95 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 81/630 objects degraded (12.857%) 2024-04-12T22:05:26.025 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:25 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:26.026 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:25 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:05:26.026 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:25 smithi097 ceph-mon[67407]: pgmap v110: 161 pgs: 26 active+undersized+degraded, 40 active+undersized, 95 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 81/630 objects degraded (12.857%) 2024-04-12T22:05:26.026 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:25 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:26.026 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:25 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:05:26.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:25 smithi114 ceph-mon[58780]: pgmap v110: 161 pgs: 26 active+undersized+degraded, 40 active+undersized, 95 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 81/630 objects degraded (12.857%) 2024-04-12T22:05:26.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:25 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:26.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:25 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:05:26.660 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:05:26 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-3[78009]: 2024-04-12T22:05:26.399+0000 7f94f5d9a7c0 -1 osd.3 0 read_superblock omap replica is missing. 2024-04-12T22:05:26.660 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:05:26 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-3[78009]: 2024-04-12T22:05:26.429+0000 7f94f5d9a7c0 -1 osd.3 121 log_to_monitors true 2024-04-12T22:05:27.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:26 smithi097 ceph-mon[64853]: from='osd.3 [v2:172.21.15.97:6826/514428802,v1:172.21.15.97:6827/514428802]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-12T22:05:27.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:26 smithi097 ceph-mon[64853]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-12T22:05:27.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:26 smithi097 ceph-mon[67407]: from='osd.3 [v2:172.21.15.97:6826/514428802,v1:172.21.15.97:6827/514428802]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-12T22:05:27.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:26 smithi097 ceph-mon[67407]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-12T22:05:27.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:26 smithi114 ceph-mon[58780]: from='osd.3 [v2:172.21.15.97:6826/514428802,v1:172.21.15.97:6827/514428802]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-12T22:05:27.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:26 smithi114 ceph-mon[58780]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-12T22:05:27.660 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:05:27 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-3[78009]: 2024-04-12T22:05:27.230+0000 7f94e31ea700 -1 osd.3 121 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T22:05:28.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:27 smithi097 ceph-mon[67407]: pgmap v111: 161 pgs: 26 active+undersized+degraded, 40 active+undersized, 95 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 81/630 objects degraded (12.857%) 2024-04-12T22:05:28.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:27 smithi097 ceph-mon[67407]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-12T22:05:28.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:27 smithi097 ceph-mon[67407]: osdmap e124: 8 total, 7 up, 8 in 2024-04-12T22:05:28.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:27 smithi097 ceph-mon[67407]: from='osd.3 [v2:172.21.15.97:6826/514428802,v1:172.21.15.97:6827/514428802]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-12T22:05:28.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:27 smithi097 ceph-mon[67407]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-12T22:05:28.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:27 smithi097 ceph-mon[67407]: from='osd.3 ' entity='osd.3' 2024-04-12T22:05:28.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:27 smithi097 ceph-mon[64853]: pgmap v111: 161 pgs: 26 active+undersized+degraded, 40 active+undersized, 95 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 81/630 objects degraded (12.857%) 2024-04-12T22:05:28.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:27 smithi097 ceph-mon[64853]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-12T22:05:28.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:27 smithi097 ceph-mon[64853]: osdmap e124: 8 total, 7 up, 8 in 2024-04-12T22:05:28.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:27 smithi097 ceph-mon[64853]: from='osd.3 [v2:172.21.15.97:6826/514428802,v1:172.21.15.97:6827/514428802]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-12T22:05:28.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:27 smithi097 ceph-mon[64853]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-12T22:05:28.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:27 smithi097 ceph-mon[64853]: from='osd.3 ' entity='osd.3' 2024-04-12T22:05:28.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:27 smithi114 ceph-mon[58780]: pgmap v111: 161 pgs: 26 active+undersized+degraded, 40 active+undersized, 95 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 81/630 objects degraded (12.857%) 2024-04-12T22:05:28.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:27 smithi114 ceph-mon[58780]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-12T22:05:28.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:27 smithi114 ceph-mon[58780]: osdmap e124: 8 total, 7 up, 8 in 2024-04-12T22:05:28.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:27 smithi114 ceph-mon[58780]: from='osd.3 [v2:172.21.15.97:6826/514428802,v1:172.21.15.97:6827/514428802]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-12T22:05:28.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:27 smithi114 ceph-mon[58780]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-04-12T22:05:28.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:27 smithi114 ceph-mon[58780]: from='osd.3 ' entity='osd.3' 2024-04-12T22:05:28.993 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:05:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:05:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"smithi097:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"172.21.15.114:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-12T22:05:28.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:28 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:28.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:28 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:28.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:28 smithi114 ceph-mon[58780]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-12T22:05:28.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:28 smithi114 ceph-mon[58780]: osd.3 [v2:172.21.15.97:6826/514428802,v1:172.21.15.97:6827/514428802] boot 2024-04-12T22:05:28.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:28 smithi114 ceph-mon[58780]: osdmap e125: 8 total, 8 up, 8 in 2024-04-12T22:05:28.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:28 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T22:05:29.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:28 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:29.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:28 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:29.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:28 smithi097 ceph-mon[64853]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-12T22:05:29.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:28 smithi097 ceph-mon[64853]: osd.3 [v2:172.21.15.97:6826/514428802,v1:172.21.15.97:6827/514428802] boot 2024-04-12T22:05:29.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:28 smithi097 ceph-mon[64853]: osdmap e125: 8 total, 8 up, 8 in 2024-04-12T22:05:29.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:28 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T22:05:29.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:28 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:29.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:28 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:29.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:28 smithi097 ceph-mon[67407]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-12T22:05:29.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:28 smithi097 ceph-mon[67407]: osd.3 [v2:172.21.15.97:6826/514428802,v1:172.21.15.97:6827/514428802] boot 2024-04-12T22:05:29.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:28 smithi097 ceph-mon[67407]: osdmap e125: 8 total, 8 up, 8 in 2024-04-12T22:05:29.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:28 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T22:05:29.470 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:05:29 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:05:29] "GET /metrics HTTP/1.1" 200 38442 "" "Prometheus/2.33.4" 2024-04-12T22:05:30.147 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:29 smithi097 ceph-mon[64853]: pgmap v113: 161 pgs: 26 active+undersized+degraded, 40 active+undersized, 95 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 81/630 objects degraded (12.857%) 2024-04-12T22:05:30.147 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:29 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:30.147 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:29 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:30.147 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:29 smithi097 ceph-mon[64853]: osdmap e126: 8 total, 8 up, 8 in 2024-04-12T22:05:30.147 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:29 smithi097 ceph-mon[67407]: pgmap v113: 161 pgs: 26 active+undersized+degraded, 40 active+undersized, 95 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 81/630 objects degraded (12.857%) 2024-04-12T22:05:30.147 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:29 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:30.147 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:29 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:30.148 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:29 smithi097 ceph-mon[67407]: osdmap e126: 8 total, 8 up, 8 in 2024-04-12T22:05:30.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:29 smithi114 ceph-mon[58780]: pgmap v113: 161 pgs: 26 active+undersized+degraded, 40 active+undersized, 95 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 81/630 objects degraded (12.857%) 2024-04-12T22:05:30.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:29 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:30.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:29 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:30.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:29 smithi114 ceph-mon[58780]: osdmap e126: 8 total, 8 up, 8 in 2024-04-12T22:05:32.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:31 smithi097 ceph-mon[64853]: pgmap v116: 161 pgs: 26 active+undersized+degraded, 40 active+undersized, 95 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 81/630 objects degraded (12.857%) 2024-04-12T22:05:32.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:31 smithi097 ceph-mon[67407]: pgmap v116: 161 pgs: 26 active+undersized+degraded, 40 active+undersized, 95 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 81/630 objects degraded (12.857%) 2024-04-12T22:05:32.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:31 smithi114 ceph-mon[58780]: pgmap v116: 161 pgs: 26 active+undersized+degraded, 40 active+undersized, 95 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 81/630 objects degraded (12.857%) 2024-04-12T22:05:33.156 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:32 smithi114 ceph-mon[58780]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:05:33.156 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:32 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:33.156 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:32 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:33.156 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:32 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:05:33.156 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:32 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:05:33.156 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:32 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:33.156 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:32 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:05:33.156 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:32 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:05:33.157 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:32 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:05:33.157 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:32 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:05:33.157 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:32 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-12T22:05:33.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:32 smithi097 ceph-mon[67407]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:05:33.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:32 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:33.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:32 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:33.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:32 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:05:33.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:32 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:05:33.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:32 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:33.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:32 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:05:33.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:32 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:05:33.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:32 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:05:33.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:32 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:05:33.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:32 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-12T22:05:33.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:32 smithi097 ceph-mon[64853]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:05:33.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:32 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:33.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:32 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:33.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:32 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:05:33.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:32 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:05:33.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:32 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:33.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:32 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:05:33.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:32 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:05:33.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:32 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:05:33.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:32 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:05:33.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:32 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-12T22:05:33.702 INFO:teuthology.orchestra.run.smithi097.stdout:true 2024-04-12T22:05:34.153 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:33 smithi114 ceph-mon[58780]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:05:34.153 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:33 smithi114 ceph-mon[58780]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:05:34.153 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:33 smithi114 ceph-mon[58780]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:05:34.153 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:33 smithi114 ceph-mon[58780]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-12T22:05:34.153 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:33 smithi114 ceph-mon[58780]: Upgrade: osd.4 is safe to restart 2024-04-12T22:05:34.153 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:33 smithi114 ceph-mon[58780]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 81/630 objects degraded (12.857%), 26 pgs degraded) 2024-04-12T22:05:34.153 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:33 smithi114 ceph-mon[58780]: from='client.44266 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:05:34.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:33 smithi097 ceph-mon[67407]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:05:34.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:33 smithi097 ceph-mon[67407]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:05:34.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:33 smithi097 ceph-mon[67407]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:05:34.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:33 smithi097 ceph-mon[67407]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-12T22:05:34.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:33 smithi097 ceph-mon[67407]: Upgrade: osd.4 is safe to restart 2024-04-12T22:05:34.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:33 smithi097 ceph-mon[67407]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 81/630 objects degraded (12.857%), 26 pgs degraded) 2024-04-12T22:05:34.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:33 smithi097 ceph-mon[67407]: from='client.44266 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:05:34.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:33 smithi097 ceph-mon[64853]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:05:34.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:33 smithi097 ceph-mon[64853]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:05:34.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:33 smithi097 ceph-mon[64853]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:05:34.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:33 smithi097 ceph-mon[64853]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-12T22:05:34.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:33 smithi097 ceph-mon[64853]: Upgrade: osd.4 is safe to restart 2024-04-12T22:05:34.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:33 smithi097 ceph-mon[64853]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 81/630 objects degraded (12.857%), 26 pgs degraded) 2024-04-12T22:05:34.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:33 smithi097 ceph-mon[64853]: from='client.44266 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:05:34.352 INFO:teuthology.orchestra.run.smithi097.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T22:05:34.353 INFO:teuthology.orchestra.run.smithi097.stdout:alertmanager.a smithi097 *:9093,9094 running (4m) 6s ago 9m 25.5M - 0.23.0 ba2b418f427c 7b4c491fe0a0 2024-04-12T22:05:34.353 INFO:teuthology.orchestra.run.smithi097.stdout:grafana.a smithi114 *:3000 unknown 2m ago 9m - - 2024-04-12T22:05:34.353 INFO:teuthology.orchestra.run.smithi097.stdout:iscsi.foo.smithi097.xndkxy smithi097 running (3m) 6s ago 8m 52.6M - 3.5 e1d6a67b021e 0436463f28cd 2024-04-12T22:05:34.353 INFO:teuthology.orchestra.run.smithi097.stdout:mgr.x smithi114 *:8443,9283,8765 running (3m) 2m ago 13m 450M - 19.0.0-2053-gb277feb2 668895944f0e b8f8df3bf928 2024-04-12T22:05:34.353 INFO:teuthology.orchestra.run.smithi097.stdout:mgr.y smithi097 *:8443,9283,8765 running (3m) 6s ago 15m 506M - 19.0.0-2053-gb277feb2 668895944f0e 488170dac11f 2024-04-12T22:05:34.353 INFO:teuthology.orchestra.run.smithi097.stdout:mon.a smithi097 running (3m) 6s ago 15m 50.6M 2048M 19.0.0-2053-gb277feb2 668895944f0e 3cfa16d3a85c 2024-04-12T22:05:34.353 INFO:teuthology.orchestra.run.smithi097.stdout:mon.b smithi114 running (2m) 2m ago 14m 18.4M 2048M 19.0.0-2053-gb277feb2 668895944f0e 71a2eff159a3 2024-04-12T22:05:34.353 INFO:teuthology.orchestra.run.smithi097.stdout:mon.c smithi097 running (2m) 6s ago 14m 39.3M 2048M 19.0.0-2053-gb277feb2 668895944f0e 69507ba88603 2024-04-12T22:05:34.353 INFO:teuthology.orchestra.run.smithi097.stdout:node-exporter.a smithi097 *:9100 running (4m) 6s ago 10m 17.7M - 1.3.1 1dbe0e931976 5edff9053b37 2024-04-12T22:05:34.353 INFO:teuthology.orchestra.run.smithi097.stdout:node-exporter.b smithi114 *:9100 running (4m) 2m ago 10m 17.0M - 1.3.1 1dbe0e931976 904afc45dc2f 2024-04-12T22:05:34.353 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0 smithi097 running (87s) 6s ago 13m 51.2M 2736M 19.0.0-2053-gb277feb2 668895944f0e 9d05fea4bf37 2024-04-12T22:05:34.353 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1 smithi097 running (66s) 6s ago 13m 43.7M 2736M 19.0.0-2053-gb277feb2 668895944f0e a9a527642889 2024-04-12T22:05:34.354 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2 smithi097 running (31s) 6s ago 12m 43.7M 2736M 19.0.0-2053-gb277feb2 668895944f0e 2e2297afff50 2024-04-12T22:05:34.354 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3 smithi097 running (11s) 6s ago 12m 12.6M 2736M 19.0.0-2053-gb277feb2 668895944f0e 415b4d3d85f2 2024-04-12T22:05:34.354 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4 smithi114 running (12m) 2m ago 12m 53.6M 2992M 17.2.0 e1d6a67b021e fb42c0e8e575 2024-04-12T22:05:34.354 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5 smithi114 running (11m) 2m ago 11m 50.2M 2992M 17.2.0 e1d6a67b021e 3acd4a3b61e5 2024-04-12T22:05:34.354 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6 smithi114 running (11m) 2m ago 11m 49.6M 2992M 17.2.0 e1d6a67b021e 5969baf67f41 2024-04-12T22:05:34.354 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7 smithi114 running (10m) 2m ago 10m 51.8M 2992M 17.2.0 e1d6a67b021e f68340fecf2c 2024-04-12T22:05:34.354 INFO:teuthology.orchestra.run.smithi097.stdout:prometheus.a smithi114 *:9095 running (3m) 2m ago 10m 65.1M - 2.33.4 514e6a882f6e 0ad5437ed63f 2024-04-12T22:05:34.354 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.foo.smithi097.sggtkh smithi097 *:8000 running (8m) 6s ago 8m 73.0M - 17.2.0 e1d6a67b021e a39f9b2d8509 2024-04-12T22:05:34.354 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.foo.smithi114.ulfbyo smithi114 *:8000 running (8m) 2m ago 8m 72.8M - 17.2.0 e1d6a67b021e df79a48bdd59 2024-04-12T22:05:34.354 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.smpl.smithi097.exxkgl smithi097 *:80 running (8m) 6s ago 8m 72.9M - 17.2.0 e1d6a67b021e a036c58a0099 2024-04-12T22:05:34.354 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.smpl.smithi114.lrgvph smithi114 *:80 running (8m) 2m ago 8m 74.4M - 17.2.0 e1d6a67b021e 031a5fc24237 2024-04-12T22:05:34.742 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-12T22:05:34.742 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": { 2024-04-12T22:05:34.742 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 3 2024-04-12T22:05:34.742 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:05:34.742 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": { 2024-04-12T22:05:34.742 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 2 2024-04-12T22:05:34.742 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:05:34.742 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": { 2024-04-12T22:05:34.743 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-04-12T22:05:34.743 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 4 2024-04-12T22:05:34.743 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:05:34.743 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw": { 2024-04-12T22:05:34.743 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-12T22:05:34.743 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:05:34.743 INFO:teuthology.orchestra.run.smithi097.stdout: "overall": { 2024-04-12T22:05:34.743 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8, 2024-04-12T22:05:34.743 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 9 2024-04-12T22:05:34.743 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-12T22:05:34.743 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-12T22:05:35.070 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-12T22:05:35.071 INFO:teuthology.orchestra.run.smithi097.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a", 2024-04-12T22:05:35.071 INFO:teuthology.orchestra.run.smithi097.stdout: "in_progress": true, 2024-04-12T22:05:35.071 INFO:teuthology.orchestra.run.smithi097.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T22:05:35.071 INFO:teuthology.orchestra.run.smithi097.stdout: "services_complete": [ 2024-04-12T22:05:35.071 INFO:teuthology.orchestra.run.smithi097.stdout: "mon", 2024-04-12T22:05:35.071 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr" 2024-04-12T22:05:35.071 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-04-12T22:05:35.071 INFO:teuthology.orchestra.run.smithi097.stdout: "progress": "9/23 daemons upgraded", 2024-04-12T22:05:35.071 INFO:teuthology.orchestra.run.smithi097.stdout: "message": "Currently upgrading osd daemons", 2024-04-12T22:05:35.071 INFO:teuthology.orchestra.run.smithi097.stdout: "is_paused": false 2024-04-12T22:05:35.071 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-12T22:05:35.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:35 smithi097 ceph-mon[64853]: from='client.44272 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:05:35.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:35 smithi097 ceph-mon[64853]: Upgrade: Updating osd.4 2024-04-12T22:05:35.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:35 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:35.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:35 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-12T22:05:35.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:35 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:05:35.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:35 smithi097 ceph-mon[64853]: Deploying daemon osd.4 on smithi114 2024-04-12T22:05:35.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:35 smithi097 ceph-mon[64853]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:05:35.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:35 smithi097 ceph-mon[64853]: from='client.44278 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:05:35.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:35 smithi097 ceph-mon[64853]: from='client.? 172.21.15.97:0/3217525776' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:05:35.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:35 smithi097 ceph-mon[67407]: from='client.44272 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:05:35.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:35 smithi097 ceph-mon[67407]: Upgrade: Updating osd.4 2024-04-12T22:05:35.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:35 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:35.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:35 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-12T22:05:35.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:35 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:05:35.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:35 smithi097 ceph-mon[67407]: Deploying daemon osd.4 on smithi114 2024-04-12T22:05:35.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:35 smithi097 ceph-mon[67407]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:05:35.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:35 smithi097 ceph-mon[67407]: from='client.44278 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:05:35.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:35 smithi097 ceph-mon[67407]: from='client.? 172.21.15.97:0/3217525776' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:05:35.451 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:35 smithi114 ceph-mon[58780]: from='client.44272 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:05:35.451 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:35 smithi114 ceph-mon[58780]: Upgrade: Updating osd.4 2024-04-12T22:05:35.451 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:35 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:35.451 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:35 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-12T22:05:35.451 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:35 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:05:35.451 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:35 smithi114 ceph-mon[58780]: Deploying daemon osd.4 on smithi114 2024-04-12T22:05:35.451 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:35 smithi114 ceph-mon[58780]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:05:35.451 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:35 smithi114 ceph-mon[58780]: from='client.44278 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:05:35.451 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:35 smithi114 ceph-mon[58780]: from='client.? 172.21.15.97:0/3217525776' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:05:35.465 INFO:teuthology.orchestra.run.smithi097.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-12T22:05:35.465 INFO:teuthology.orchestra.run.smithi097.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-12T22:05:35.465 INFO:teuthology.orchestra.run.smithi097.stdout: daemon grafana.a on smithi114 is in unknown state 2024-04-12T22:05:36.347 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:36 smithi114 ceph-mon[58780]: from='client.34362 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:05:36.347 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:36 smithi114 ceph-mon[58780]: from='client.? 172.21.15.97:0/1219731743' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T22:05:36.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:36 smithi097 ceph-mon[64853]: from='client.34362 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:05:36.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:36 smithi097 ceph-mon[64853]: from='client.? 172.21.15.97:0/1219731743' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T22:05:36.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:36 smithi097 ceph-mon[67407]: from='client.34362 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:05:36.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:36 smithi097 ceph-mon[67407]: from='client.? 172.21.15.97:0/1219731743' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T22:05:37.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:37 smithi114 ceph-mon[58780]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:05:37.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:37 smithi097 ceph-mon[64853]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:05:37.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:37 smithi097 ceph-mon[67407]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:05:37.864 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:05:37 smithi114 systemd[1]: Stopping Ceph osd.4 for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:05:37.864 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:05:37 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-4[29183]: 2024-04-12T22:05:37.862+0000 7f9e819b0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-12T22:05:37.864 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:05:37 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-4[29183]: 2024-04-12T22:05:37.862+0000 7f9e819b0700 -1 osd.4 126 *** Got signal Terminated *** 2024-04-12T22:05:37.864 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:05:37 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-4[29183]: 2024-04-12T22:05:37.862+0000 7f9e819b0700 -1 osd.4 126 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-12T22:05:38.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:38 smithi114 ceph-mon[58780]: osd.4 marked itself down and dead 2024-04-12T22:05:38.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:38 smithi097 ceph-mon[64853]: osd.4 marked itself down and dead 2024-04-12T22:05:38.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:38 smithi097 ceph-mon[67407]: osd.4 marked itself down and dead 2024-04-12T22:05:38.993 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:05:38 smithi114 podman[61965]: 2024-04-12 22:05:38.677815921 +0000 UTC m=+0.909029960 container died fb42c0e8e5750fb6a843fd06de1bf9a2428eb606705d6b121dfd7fd26053460c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-4, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, architecture=x86_64, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, com.redhat.component=centos-stream-container, vcs-type=git, build-date=2022-05-03T08:36:31.336870) 2024-04-12T22:05:38.993 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:05:38 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:05:38.659Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"smithi097:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"172.21.15.114:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-04-12T22:05:38.993 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:05:38 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:05:38.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"smithi097:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"172.21.15.114:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-12T22:05:39.356 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:39 smithi114 ceph-mon[58780]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-12T22:05:39.356 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:39 smithi114 ceph-mon[58780]: osdmap e127: 8 total, 7 up, 8 in 2024-04-12T22:05:39.356 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:39 smithi114 ceph-mon[58780]: pgmap v123: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 840 B/s rd, 0 op/s 2024-04-12T22:05:39.357 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:05:39 smithi114 podman[61965]: 2024-04-12 22:05:39.079749701 +0000 UTC m=+1.310963721 container cleanup fb42c0e8e5750fb6a843fd06de1bf9a2428eb606705d6b121dfd7fd26053460c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-4, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, release=754, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , io.openshift.expose-services=, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, distribution-scope=public, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-04-12T22:05:39.357 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:05:39 smithi114 bash[61965]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-4 2024-04-12T22:05:39.660 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:05:39 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:05:39] "GET /metrics HTTP/1.1" 200 38442 "" "Prometheus/2.33.4" 2024-04-12T22:05:39.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:39 smithi097 ceph-mon[64853]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-12T22:05:39.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:39 smithi097 ceph-mon[64853]: osdmap e127: 8 total, 7 up, 8 in 2024-04-12T22:05:39.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:39 smithi097 ceph-mon[64853]: pgmap v123: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 840 B/s rd, 0 op/s 2024-04-12T22:05:39.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:39 smithi097 ceph-mon[67407]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-12T22:05:39.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:39 smithi097 ceph-mon[67407]: osdmap e127: 8 total, 7 up, 8 in 2024-04-12T22:05:39.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:39 smithi097 ceph-mon[67407]: pgmap v123: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 840 B/s rd, 0 op/s 2024-04-12T22:05:39.743 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:05:39 smithi114 podman[61994]: 2024-04-12 22:05:39.356066633 +0000 UTC m=+0.678313042 container remove fb42c0e8e5750fb6a843fd06de1bf9a2428eb606705d6b121dfd7fd26053460c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-4, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, RELEASE=HEAD, version=8, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, distribution-scope=public, GIT_CLEAN=True, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-12T22:05:40.009 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:05:39 smithi114 podman[62089]: 2024-04-12 22:05:39.655385708 +0000 UTC m=+0.017965228 image pull 668895944f0ec64b67a5b7564961d2756f2966e5d4b9338516300dbec9984951 quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a 2024-04-12T22:05:40.009 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:05:39 smithi114 podman[62089]: 2024-04-12 22:05:39.773879534 +0000 UTC m=+0.136459052 container create cde42003eeef32671b45bc8295c049abe409dae845c02f7f19ba28ddee034136 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-b277feb, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409) 2024-04-12T22:05:40.301 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:05:40 smithi114 podman[62089]: 2024-04-12 22:05:40.009604944 +0000 UTC m=+0.372184460 container init cde42003eeef32671b45bc8295c049abe409dae845c02f7f19ba28ddee034136 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-b277feb, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-12T22:05:40.301 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:05:40 smithi114 podman[62089]: 2024-04-12 22:05:40.01284823 +0000 UTC m=+0.375427753 container start cde42003eeef32671b45bc8295c049abe409dae845c02f7f19ba28ddee034136 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-4-deactivate, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-b277feb, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS) 2024-04-12T22:05:40.301 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:05:40 smithi114 podman[62089]: 2024-04-12 22:05:40.084566059 +0000 UTC m=+0.447145584 container attach cde42003eeef32671b45bc8295c049abe409dae845c02f7f19ba28ddee034136 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-b277feb, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux ) 2024-04-12T22:05:40.301 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:40 smithi114 ceph-mon[58780]: osdmap e128: 8 total, 7 up, 8 in 2024-04-12T22:05:40.301 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:40 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 4]}]: dispatch 2024-04-12T22:05:40.301 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:40 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5, 3, 0]}]: dispatch 2024-04-12T22:05:40.301 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:40 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 4]}]: dispatch 2024-04-12T22:05:40.302 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:40 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 4]}]: dispatch 2024-04-12T22:05:40.302 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:40 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 4]}]: dispatch 2024-04-12T22:05:40.302 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:40 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 4]}]: dispatch 2024-04-12T22:05:40.302 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:40 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-04-12T22:05:40.302 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:40 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1a", "id": [3, 2]}]: dispatch 2024-04-12T22:05:40.302 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:40 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [3, 4]}]: dispatch 2024-04-12T22:05:40.302 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:40 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:40.302 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:40 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:05:40.302 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:40 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:40.579 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:05:40 smithi114 podman[62089]: 2024-04-12 22:05:40.301233094 +0000 UTC m=+0.663812611 container died cde42003eeef32671b45bc8295c049abe409dae845c02f7f19ba28ddee034136 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=squid-b277feb, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, ceph=True) 2024-04-12T22:05:40.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:40 smithi097 ceph-mon[64853]: osdmap e128: 8 total, 7 up, 8 in 2024-04-12T22:05:40.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:40 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 4]}]: dispatch 2024-04-12T22:05:40.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:40 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5, 3, 0]}]: dispatch 2024-04-12T22:05:40.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:40 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 4]}]: dispatch 2024-04-12T22:05:40.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:40 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 4]}]: dispatch 2024-04-12T22:05:40.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:40 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 4]}]: dispatch 2024-04-12T22:05:40.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:40 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 4]}]: dispatch 2024-04-12T22:05:40.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:40 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-04-12T22:05:40.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:40 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1a", "id": [3, 2]}]: dispatch 2024-04-12T22:05:40.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:40 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [3, 4]}]: dispatch 2024-04-12T22:05:40.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:40 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:40.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:40 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:05:40.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:40 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:40.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:40 smithi097 ceph-mon[67407]: osdmap e128: 8 total, 7 up, 8 in 2024-04-12T22:05:40.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:40 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 4]}]: dispatch 2024-04-12T22:05:40.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:40 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5, 3, 0]}]: dispatch 2024-04-12T22:05:40.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:40 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 4]}]: dispatch 2024-04-12T22:05:40.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:40 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 4]}]: dispatch 2024-04-12T22:05:40.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:40 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 4]}]: dispatch 2024-04-12T22:05:40.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:40 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 4]}]: dispatch 2024-04-12T22:05:40.663 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:40 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-04-12T22:05:40.663 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:40 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1a", "id": [3, 2]}]: dispatch 2024-04-12T22:05:40.663 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:40 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [3, 4]}]: dispatch 2024-04-12T22:05:40.663 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:40 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:40.663 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:40 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:05:40.663 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:40 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:41.243 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:05:40 smithi114 podman[62116]: 2024-04-12 22:05:40.971752589 +0000 UTC m=+0.663229099 container remove cde42003eeef32671b45bc8295c049abe409dae845c02f7f19ba28ddee034136 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-b277feb, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-12T22:05:41.243 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:05:40 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@osd.4.service: Deactivated successfully. 2024-04-12T22:05:41.243 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:05:40 smithi114 systemd[1]: Stopped Ceph osd.4 for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:05:41.243 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:05:40 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@osd.4.service: Consumed 8.474s CPU time. 2024-04-12T22:05:41.243 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:05:41 smithi114 systemd[1]: Starting Ceph osd.4 for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:05:41.552 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:41 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 4]}]': finished 2024-04-12T22:05:41.552 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:41 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5, 3, 0]}]': finished 2024-04-12T22:05:41.553 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:41 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 4]}]': finished 2024-04-12T22:05:41.553 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:41 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 4]}]': finished 2024-04-12T22:05:41.553 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:41 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 4]}]': finished 2024-04-12T22:05:41.553 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:41 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 4]}]': finished 2024-04-12T22:05:41.553 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:41 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-04-12T22:05:41.553 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:41 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1a", "id": [3, 2]}]': finished 2024-04-12T22:05:41.553 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:41 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [3, 4]}]': finished 2024-04-12T22:05:41.553 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:41 smithi114 ceph-mon[58780]: osdmap e129: 8 total, 7 up, 8 in 2024-04-12T22:05:41.553 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:41 smithi114 ceph-mon[58780]: pgmap v126: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:05:41.553 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:05:41 smithi114 podman[62220]: 2024-04-12 22:05:41.320342177 +0000 UTC m=+0.017910539 image pull 668895944f0ec64b67a5b7564961d2756f2966e5d4b9338516300dbec9984951 quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a 2024-04-12T22:05:41.553 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:05:41 smithi114 podman[62220]: 2024-04-12 22:05:41.439371573 +0000 UTC m=+0.136939934 container create ff9e3f58fa463d927c0d0a35a304b96859ecf6c9fb1ea9cceacc0c69bc62e7f1 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-4-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b277feb, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-12T22:05:41.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:41 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 4]}]': finished 2024-04-12T22:05:41.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:41 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5, 3, 0]}]': finished 2024-04-12T22:05:41.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:41 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 4]}]': finished 2024-04-12T22:05:41.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:41 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 4]}]': finished 2024-04-12T22:05:41.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:41 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 4]}]': finished 2024-04-12T22:05:41.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:41 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 4]}]': finished 2024-04-12T22:05:41.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:41 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-04-12T22:05:41.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:41 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1a", "id": [3, 2]}]': finished 2024-04-12T22:05:41.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:41 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [3, 4]}]': finished 2024-04-12T22:05:41.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:41 smithi097 ceph-mon[64853]: osdmap e129: 8 total, 7 up, 8 in 2024-04-12T22:05:41.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:41 smithi097 ceph-mon[64853]: pgmap v126: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:05:41.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:41 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 4]}]': finished 2024-04-12T22:05:41.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:41 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5, 3, 0]}]': finished 2024-04-12T22:05:41.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:41 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 4]}]': finished 2024-04-12T22:05:41.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:41 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 4]}]': finished 2024-04-12T22:05:41.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:41 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 4]}]': finished 2024-04-12T22:05:41.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:41 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 4]}]': finished 2024-04-12T22:05:41.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:41 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-04-12T22:05:41.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:41 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1a", "id": [3, 2]}]': finished 2024-04-12T22:05:41.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:41 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [3, 4]}]': finished 2024-04-12T22:05:41.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:41 smithi097 ceph-mon[67407]: osdmap e129: 8 total, 7 up, 8 in 2024-04-12T22:05:41.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:41 smithi097 ceph-mon[67407]: pgmap v126: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:05:41.993 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:05:41 smithi114 podman[62220]: 2024-04-12 22:05:41.6667639 +0000 UTC m=+0.364332262 container init ff9e3f58fa463d927c0d0a35a304b96859ecf6c9fb1ea9cceacc0c69bc62e7f1 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-4-activate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-b277feb, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-12T22:05:41.993 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:05:41 smithi114 podman[62220]: 2024-04-12 22:05:41.669835551 +0000 UTC m=+0.367403913 container start ff9e3f58fa463d927c0d0a35a304b96859ecf6c9fb1ea9cceacc0c69bc62e7f1 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-4-activate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-b277feb, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-12T22:05:41.993 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:05:41 smithi114 podman[62220]: 2024-04-12 22:05:41.741622643 +0000 UTC m=+0.439191021 container attach ff9e3f58fa463d927c0d0a35a304b96859ecf6c9fb1ea9cceacc0c69bc62e7f1 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-4-activate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-b277feb, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-12T22:05:42.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:42 smithi114 ceph-mon[58780]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:05:42.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:42 smithi114 ceph-mon[58780]: osdmap e130: 8 total, 7 up, 8 in 2024-04-12T22:05:42.493 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:05:42 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-4-activate[62236]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-12T22:05:42.493 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:05:42 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-4-activate[62236]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-12T22:05:42.493 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:05:42 smithi114 bash[62220]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-12T22:05:42.493 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:05:42 smithi114 bash[62220]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-12T22:05:42.493 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:05:42 smithi114 bash[62220]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-12T22:05:42.494 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:05:42 smithi114 bash[62220]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-12T22:05:42.494 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:05:42 smithi114 bash[62220]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-04-12T22:05:42.494 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:05:42 smithi114 bash[62220]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-12T22:05:42.494 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:05:42 smithi114 bash[62220]: --> ceph-volume raw activate successful for osd ID: 4 2024-04-12T22:05:42.494 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:05:42 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-4-activate[62236]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-12T22:05:42.494 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:05:42 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-4-activate[62236]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-12T22:05:42.494 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:05:42 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-4-activate[62236]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-04-12T22:05:42.494 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:05:42 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-4-activate[62236]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-12T22:05:42.494 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:05:42 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-4-activate[62236]: --> ceph-volume raw activate successful for osd ID: 4 2024-04-12T22:05:42.494 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:05:42 smithi114 podman[62220]: 2024-04-12 22:05:42.133374484 +0000 UTC m=+0.830942842 container died ff9e3f58fa463d927c0d0a35a304b96859ecf6c9fb1ea9cceacc0c69bc62e7f1 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-4-activate, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b277feb, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-12T22:05:42.595 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:42 smithi097 ceph-mon[64853]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:05:42.595 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:42 smithi097 ceph-mon[64853]: osdmap e130: 8 total, 7 up, 8 in 2024-04-12T22:05:42.595 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:42 smithi097 ceph-mon[67407]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:05:42.595 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:42 smithi097 ceph-mon[67407]: osdmap e130: 8 total, 7 up, 8 in 2024-04-12T22:05:43.086 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:05:42 smithi114 podman[62358]: 2024-04-12 22:05:42.755985149 +0000 UTC m=+0.613151217 container remove ff9e3f58fa463d927c0d0a35a304b96859ecf6c9fb1ea9cceacc0c69bc62e7f1 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-4-activate, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, RELEASE=squid-b277feb, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-12T22:05:43.087 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:05:42 smithi114 podman[62423]: 2024-04-12 22:05:42.86948517 +0000 UTC m=+0.017791389 image pull 668895944f0ec64b67a5b7564961d2756f2966e5d4b9338516300dbec9984951 quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a 2024-04-12T22:05:43.087 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:05:42 smithi114 podman[62423]: 2024-04-12 22:05:42.973485327 +0000 UTC m=+0.121791564 container create fcd2e39a5866e206a9536540cb168f9f44cbd284636e137f6b79e12bae0da4b6 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-4, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=squid-b277feb, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-12T22:05:43.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:43 smithi114 ceph-mon[58780]: osdmap e131: 8 total, 7 up, 8 in 2024-04-12T22:05:43.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:43 smithi114 ceph-mon[58780]: pgmap v129: 161 pgs: 2 remapped+peering, 5 peering, 28 active+undersized+degraded, 1 stale+active+clean, 38 active+undersized, 87 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 108/630 objects degraded (17.143%); 0 B/s, 0 objects/s recovering 2024-04-12T22:05:43.493 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:05:43 smithi114 podman[62423]: 2024-04-12 22:05:43.200703845 +0000 UTC m=+0.349010040 container init fcd2e39a5866e206a9536540cb168f9f44cbd284636e137f6b79e12bae0da4b6 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-4, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-b277feb, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-12T22:05:43.493 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:05:43 smithi114 podman[62423]: 2024-04-12 22:05:43.203767546 +0000 UTC m=+0.352073741 container start fcd2e39a5866e206a9536540cb168f9f44cbd284636e137f6b79e12bae0da4b6 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-4, RELEASE=squid-b277feb, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409) 2024-04-12T22:05:43.493 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:05:43 smithi114 ceph-osd[62442]: -- 172.21.15.114:0/2958399935 <== mon.2 v2:172.21.15.114:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5629376cf680 con 0x562937699000 2024-04-12T22:05:43.493 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:05:43 smithi114 bash[62423]: fcd2e39a5866e206a9536540cb168f9f44cbd284636e137f6b79e12bae0da4b6 2024-04-12T22:05:43.494 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:05:43 smithi114 systemd[1]: Started Ceph osd.4 for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:05:43.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:43 smithi097 ceph-mon[64853]: osdmap e131: 8 total, 7 up, 8 in 2024-04-12T22:05:43.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:43 smithi097 ceph-mon[64853]: pgmap v129: 161 pgs: 2 remapped+peering, 5 peering, 28 active+undersized+degraded, 1 stale+active+clean, 38 active+undersized, 87 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 108/630 objects degraded (17.143%); 0 B/s, 0 objects/s recovering 2024-04-12T22:05:43.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:43 smithi097 ceph-mon[67407]: osdmap e131: 8 total, 7 up, 8 in 2024-04-12T22:05:43.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:43 smithi097 ceph-mon[67407]: pgmap v129: 161 pgs: 2 remapped+peering, 5 peering, 28 active+undersized+degraded, 1 stale+active+clean, 38 active+undersized, 87 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 108/630 objects degraded (17.143%); 0 B/s, 0 objects/s recovering 2024-04-12T22:05:44.131 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:05:43 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-4[62437]: 2024-04-12T22:05:43.757+0000 7f62314e07c0 -1 Falling back to public interface 2024-04-12T22:05:44.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:44 smithi114 ceph-mon[58780]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-12T22:05:44.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:44 smithi114 ceph-mon[58780]: Health check failed: Degraded data redundancy: 108/630 objects degraded (17.143%), 28 pgs degraded (PG_DEGRADED) 2024-04-12T22:05:44.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:44 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:44.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:44 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:44.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:44 smithi097 ceph-mon[64853]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-12T22:05:44.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:44 smithi097 ceph-mon[64853]: Health check failed: Degraded data redundancy: 108/630 objects degraded (17.143%), 28 pgs degraded (PG_DEGRADED) 2024-04-12T22:05:44.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:44 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:44.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:44 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:44.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:44 smithi097 ceph-mon[67407]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-12T22:05:44.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:44 smithi097 ceph-mon[67407]: Health check failed: Degraded data redundancy: 108/630 objects degraded (17.143%), 28 pgs degraded (PG_DEGRADED) 2024-04-12T22:05:44.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:44 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:44.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:44 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:45.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:45 smithi097 ceph-mon[64853]: pgmap v130: 161 pgs: 2 remapped+peering, 5 peering, 28 active+undersized+degraded, 1 stale+active+clean, 38 active+undersized, 87 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 108/630 objects degraded (17.143%); 0 B/s, 0 objects/s recovering 2024-04-12T22:05:45.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:45 smithi097 ceph-mon[67407]: pgmap v130: 161 pgs: 2 remapped+peering, 5 peering, 28 active+undersized+degraded, 1 stale+active+clean, 38 active+undersized, 87 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 108/630 objects degraded (17.143%); 0 B/s, 0 objects/s recovering 2024-04-12T22:05:45.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:45 smithi114 ceph-mon[58780]: pgmap v130: 161 pgs: 2 remapped+peering, 5 peering, 28 active+undersized+degraded, 1 stale+active+clean, 38 active+undersized, 87 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 108/630 objects degraded (17.143%); 0 B/s, 0 objects/s recovering 2024-04-12T22:05:46.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:46 smithi097 ceph-mon[64853]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-04-12T22:05:46.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:46 smithi097 ceph-mon[67407]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-04-12T22:05:46.681 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:46 smithi114 ceph-mon[58780]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-04-12T22:05:46.681 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:05:46 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-4[62437]: 2024-04-12T22:05:46.460+0000 7f62314e07c0 -1 osd.4 0 read_superblock omap replica is missing. 2024-04-12T22:05:46.682 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:05:46 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-4[62437]: 2024-04-12T22:05:46.490+0000 7f62314e07c0 -1 osd.4 126 log_to_monitors true 2024-04-12T22:05:47.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:47 smithi097 ceph-mon[64853]: pgmap v131: 161 pgs: 6 active+clean+remapped, 87 active+clean, 38 active+undersized, 28 active+undersized+degraded, 2 remapped+peering; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 108/630 objects degraded (17.143%); 2/630 objects misplaced (0.317%); 0 B/s, 0 objects/s recovering 2024-04-12T22:05:47.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:47 smithi097 ceph-mon[64853]: from='osd.4 [v2:172.21.15.114:6800/2118493291,v1:172.21.15.114:6801/2118493291]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-12T22:05:47.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:47 smithi097 ceph-mon[64853]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-12T22:05:47.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:47 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:47.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:47 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:47.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:47 smithi097 ceph-mon[67407]: pgmap v131: 161 pgs: 6 active+clean+remapped, 87 active+clean, 38 active+undersized, 28 active+undersized+degraded, 2 remapped+peering; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 108/630 objects degraded (17.143%); 2/630 objects misplaced (0.317%); 0 B/s, 0 objects/s recovering 2024-04-12T22:05:47.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:47 smithi097 ceph-mon[67407]: from='osd.4 [v2:172.21.15.114:6800/2118493291,v1:172.21.15.114:6801/2118493291]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-12T22:05:47.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:47 smithi097 ceph-mon[67407]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-12T22:05:47.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:47 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:47.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:47 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:47.743 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:05:47 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-4[62437]: 2024-04-12T22:05:47.429+0000 7f6222938700 -1 osd.4 126 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T22:05:47.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:47 smithi114 ceph-mon[58780]: pgmap v131: 161 pgs: 6 active+clean+remapped, 87 active+clean, 38 active+undersized, 28 active+undersized+degraded, 2 remapped+peering; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 108/630 objects degraded (17.143%); 2/630 objects misplaced (0.317%); 0 B/s, 0 objects/s recovering 2024-04-12T22:05:47.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:47 smithi114 ceph-mon[58780]: from='osd.4 [v2:172.21.15.114:6800/2118493291,v1:172.21.15.114:6801/2118493291]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-12T22:05:47.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:47 smithi114 ceph-mon[58780]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-12T22:05:47.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:47 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:47.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:47 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:48.659 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:48 smithi114 ceph-mon[58780]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-12T22:05:48.659 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:48 smithi114 ceph-mon[58780]: osdmap e132: 8 total, 7 up, 8 in 2024-04-12T22:05:48.659 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:48 smithi114 ceph-mon[58780]: from='osd.4 [v2:172.21.15.114:6800/2118493291,v1:172.21.15.114:6801/2118493291]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-12T22:05:48.659 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:48 smithi114 ceph-mon[58780]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-12T22:05:48.659 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:48 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:48.659 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:48 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:48.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:48 smithi097 ceph-mon[64853]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-12T22:05:48.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:48 smithi097 ceph-mon[64853]: osdmap e132: 8 total, 7 up, 8 in 2024-04-12T22:05:48.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:48 smithi097 ceph-mon[64853]: from='osd.4 [v2:172.21.15.114:6800/2118493291,v1:172.21.15.114:6801/2118493291]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-12T22:05:48.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:48 smithi097 ceph-mon[64853]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-12T22:05:48.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:48 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:48.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:48 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:48.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:48 smithi097 ceph-mon[67407]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-12T22:05:48.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:48 smithi097 ceph-mon[67407]: osdmap e132: 8 total, 7 up, 8 in 2024-04-12T22:05:48.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:48 smithi097 ceph-mon[67407]: from='osd.4 [v2:172.21.15.114:6800/2118493291,v1:172.21.15.114:6801/2118493291]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-12T22:05:48.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:48 smithi097 ceph-mon[67407]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-12T22:05:48.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:48 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:48.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:48 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:48.971 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:05:48 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:05:48.658Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"smithi097:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"172.21.15.114:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-04-12T22:05:48.971 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:05:48 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:05:48.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"smithi097:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"172.21.15.114:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-12T22:05:49.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:49 smithi114 ceph-mon[58780]: pgmap v133: 161 pgs: 8 active+clean+remapped, 87 active+clean, 38 active+undersized, 28 active+undersized+degraded; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 108/630 objects degraded (17.143%); 2/630 objects misplaced (0.317%); 0 B/s, 0 objects/s recovering 2024-04-12T22:05:49.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:49 smithi114 ceph-mon[58780]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-12T22:05:49.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:49 smithi114 ceph-mon[58780]: osd.4 [v2:172.21.15.114:6800/2118493291,v1:172.21.15.114:6801/2118493291] boot 2024-04-12T22:05:49.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:49 smithi114 ceph-mon[58780]: osdmap e133: 8 total, 8 up, 8 in 2024-04-12T22:05:49.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:49 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T22:05:49.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:49 smithi114 ceph-mon[58780]: osdmap e134: 8 total, 8 up, 8 in 2024-04-12T22:05:49.660 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:05:49 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:05:49] "GET /metrics HTTP/1.1" 200 38451 "" "Prometheus/2.33.4" 2024-04-12T22:05:49.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:49 smithi097 ceph-mon[64853]: pgmap v133: 161 pgs: 8 active+clean+remapped, 87 active+clean, 38 active+undersized, 28 active+undersized+degraded; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 108/630 objects degraded (17.143%); 2/630 objects misplaced (0.317%); 0 B/s, 0 objects/s recovering 2024-04-12T22:05:49.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:49 smithi097 ceph-mon[64853]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-12T22:05:49.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:49 smithi097 ceph-mon[64853]: osd.4 [v2:172.21.15.114:6800/2118493291,v1:172.21.15.114:6801/2118493291] boot 2024-04-12T22:05:49.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:49 smithi097 ceph-mon[64853]: osdmap e133: 8 total, 8 up, 8 in 2024-04-12T22:05:49.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:49 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T22:05:49.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:49 smithi097 ceph-mon[64853]: osdmap e134: 8 total, 8 up, 8 in 2024-04-12T22:05:49.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:49 smithi097 ceph-mon[67407]: pgmap v133: 161 pgs: 8 active+clean+remapped, 87 active+clean, 38 active+undersized, 28 active+undersized+degraded; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 108/630 objects degraded (17.143%); 2/630 objects misplaced (0.317%); 0 B/s, 0 objects/s recovering 2024-04-12T22:05:49.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:49 smithi097 ceph-mon[67407]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-12T22:05:49.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:49 smithi097 ceph-mon[67407]: osd.4 [v2:172.21.15.114:6800/2118493291,v1:172.21.15.114:6801/2118493291] boot 2024-04-12T22:05:49.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:49 smithi097 ceph-mon[67407]: osdmap e133: 8 total, 8 up, 8 in 2024-04-12T22:05:49.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:49 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T22:05:49.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:49 smithi097 ceph-mon[67407]: osdmap e134: 8 total, 8 up, 8 in 2024-04-12T22:05:51.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:51 smithi114 ceph-mon[58780]: pgmap v136: 161 pgs: 8 active+clean+remapped, 87 active+clean, 38 active+undersized, 28 active+undersized+degraded; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 108/630 objects degraded (17.143%); 2/630 objects misplaced (0.317%); 0 B/s, 0 objects/s recovering 2024-04-12T22:05:51.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:51 smithi114 ceph-mon[58780]: osdmap e135: 8 total, 8 up, 8 in 2024-04-12T22:05:51.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:51 smithi097 ceph-mon[64853]: pgmap v136: 161 pgs: 8 active+clean+remapped, 87 active+clean, 38 active+undersized, 28 active+undersized+degraded; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 108/630 objects degraded (17.143%); 2/630 objects misplaced (0.317%); 0 B/s, 0 objects/s recovering 2024-04-12T22:05:51.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:51 smithi097 ceph-mon[64853]: osdmap e135: 8 total, 8 up, 8 in 2024-04-12T22:05:51.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:51 smithi097 ceph-mon[67407]: pgmap v136: 161 pgs: 8 active+clean+remapped, 87 active+clean, 38 active+undersized, 28 active+undersized+degraded; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 108/630 objects degraded (17.143%); 2/630 objects misplaced (0.317%); 0 B/s, 0 objects/s recovering 2024-04-12T22:05:51.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:51 smithi097 ceph-mon[67407]: osdmap e135: 8 total, 8 up, 8 in 2024-04-12T22:05:52.409 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:05:52 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:05:52.039+0000 7fe40fa5a700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-04-12T22:05:52.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:52 smithi097 ceph-mon[67407]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:05:52.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:52 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:52.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:52 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:52.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:52 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:05:52.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:52 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:05:52.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:52 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:52.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:52 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:05:52.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:52 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:05:52.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:52 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:05:52.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:52 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:05:52.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:52 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-12T22:05:52.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:52 smithi097 ceph-mon[64853]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:05:52.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:52 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:52.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:52 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:52.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:52 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:05:52.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:52 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:05:52.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:52 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:52.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:52 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:05:52.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:52 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:05:52.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:52 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:05:52.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:52 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:05:52.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:52 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-12T22:05:52.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:52 smithi114 ceph-mon[58780]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:05:52.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:52 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:52.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:52 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:52.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:52 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:05:52.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:52 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:05:52.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:52 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:52.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:52 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:05:52.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:52 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:05:52.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:52 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:05:52.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:52 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:05:52.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:52 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-12T22:05:53.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:53 smithi097 ceph-mon[67407]: pgmap v138: 161 pgs: 5 active+clean+remapped, 124 active+clean, 21 active+undersized, 11 active+undersized+degraded; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 35/630 objects degraded (5.556%); 2/630 objects misplaced (0.317%) 2024-04-12T22:05:53.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:53 smithi097 ceph-mon[67407]: pgmap v139: 161 pgs: 5 active+clean+remapped, 124 active+clean, 21 active+undersized, 11 active+undersized+degraded; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 35/630 objects degraded (5.556%); 2/630 objects misplaced (0.317%) 2024-04-12T22:05:53.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:53 smithi097 ceph-mon[67407]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-12T22:05:53.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:53 smithi097 ceph-mon[67407]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-04-12T22:05:53.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:53 smithi097 ceph-mon[67407]: Health check update: Degraded data redundancy: 35/630 objects degraded (5.556%), 11 pgs degraded (PG_DEGRADED) 2024-04-12T22:05:53.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:53 smithi097 ceph-mon[64853]: pgmap v138: 161 pgs: 5 active+clean+remapped, 124 active+clean, 21 active+undersized, 11 active+undersized+degraded; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 35/630 objects degraded (5.556%); 2/630 objects misplaced (0.317%) 2024-04-12T22:05:53.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:53 smithi097 ceph-mon[64853]: pgmap v139: 161 pgs: 5 active+clean+remapped, 124 active+clean, 21 active+undersized, 11 active+undersized+degraded; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 35/630 objects degraded (5.556%); 2/630 objects misplaced (0.317%) 2024-04-12T22:05:53.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:53 smithi097 ceph-mon[64853]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-12T22:05:53.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:53 smithi097 ceph-mon[64853]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-04-12T22:05:53.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:53 smithi097 ceph-mon[64853]: Health check update: Degraded data redundancy: 35/630 objects degraded (5.556%), 11 pgs degraded (PG_DEGRADED) 2024-04-12T22:05:53.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:53 smithi114 ceph-mon[58780]: pgmap v138: 161 pgs: 5 active+clean+remapped, 124 active+clean, 21 active+undersized, 11 active+undersized+degraded; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 35/630 objects degraded (5.556%); 2/630 objects misplaced (0.317%) 2024-04-12T22:05:53.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:53 smithi114 ceph-mon[58780]: pgmap v139: 161 pgs: 5 active+clean+remapped, 124 active+clean, 21 active+undersized, 11 active+undersized+degraded; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 35/630 objects degraded (5.556%); 2/630 objects misplaced (0.317%) 2024-04-12T22:05:53.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:53 smithi114 ceph-mon[58780]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-12T22:05:53.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:53 smithi114 ceph-mon[58780]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-04-12T22:05:53.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:53 smithi114 ceph-mon[58780]: Health check update: Degraded data redundancy: 35/630 objects degraded (5.556%), 11 pgs degraded (PG_DEGRADED) 2024-04-12T22:05:54.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:54 smithi114 ceph-mon[58780]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 35/630 objects degraded (5.556%), 11 pgs degraded) 2024-04-12T22:05:54.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:54 smithi097 ceph-mon[64853]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 35/630 objects degraded (5.556%), 11 pgs degraded) 2024-04-12T22:05:54.910 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:54 smithi097 ceph-mon[67407]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 35/630 objects degraded (5.556%), 11 pgs degraded) 2024-04-12T22:05:56.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:55 smithi097 ceph-mon[64853]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 933 B/s rd, 0 op/s 2024-04-12T22:05:56.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:55 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:56.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:55 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:05:56.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:55 smithi097 ceph-mon[67407]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 933 B/s rd, 0 op/s 2024-04-12T22:05:56.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:55 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:56.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:55 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:05:56.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:55 smithi114 ceph-mon[58780]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 933 B/s rd, 0 op/s 2024-04-12T22:05:56.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:55 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:05:56.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:55 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:05:57.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:57 smithi114 ceph-mon[58780]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 789 B/s rd, 0 op/s 2024-04-12T22:05:58.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:57 smithi097 ceph-mon[67407]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 789 B/s rd, 0 op/s 2024-04-12T22:05:58.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:57 smithi097 ceph-mon[64853]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 789 B/s rd, 0 op/s 2024-04-12T22:05:58.993 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:05:58 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:05:58.659Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"smithi097:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"172.21.15.114:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-04-12T22:05:58.993 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:05:58 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:05:58.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"smithi097:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"172.21.15.114:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-12T22:05:59.660 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:05:59 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:05:59] "GET /metrics HTTP/1.1" 200 38454 "" "Prometheus/2.33.4" 2024-04-12T22:06:00.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:05:59 smithi097 ceph-mon[64853]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-12T22:06:00.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:05:59 smithi097 ceph-mon[67407]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-12T22:06:00.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:05:59 smithi114 ceph-mon[58780]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-12T22:06:01.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:00 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:01.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:00 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:01.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:00 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:02.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:01 smithi097 ceph-mon[64853]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-12T22:06:02.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:01 smithi097 ceph-mon[67407]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-12T22:06:02.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:01 smithi114 ceph-mon[58780]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-12T22:06:03.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:02 smithi097 ceph-mon[64853]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:06:03.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:02 smithi097 ceph-mon[67407]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:06:03.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:02 smithi114 ceph-mon[58780]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:06:04.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:03 smithi097 ceph-mon[64853]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-12T22:06:04.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:03 smithi097 ceph-mon[67407]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-12T22:06:04.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:03 smithi114 ceph-mon[58780]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-12T22:06:05.820 INFO:teuthology.orchestra.run.smithi097.stdout:true 2024-04-12T22:06:06.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:05 smithi097 ceph-mon[64853]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:06:06.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:05 smithi097 ceph-mon[67407]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:06:06.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:05 smithi114 ceph-mon[58780]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:06:06.467 INFO:teuthology.orchestra.run.smithi097.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T22:06:06.468 INFO:teuthology.orchestra.run.smithi097.stdout:alertmanager.a smithi097 *:9093,9094 running (5m) 38s ago 10m 25.5M - 0.23.0 ba2b418f427c 7b4c491fe0a0 2024-04-12T22:06:06.468 INFO:teuthology.orchestra.run.smithi097.stdout:grafana.a smithi114 *:3000 error 19s ago 9m - - 2024-04-12T22:06:06.468 INFO:teuthology.orchestra.run.smithi097.stdout:iscsi.foo.smithi097.xndkxy smithi097 running (4m) 38s ago 8m 52.6M - 3.5 e1d6a67b021e 0436463f28cd 2024-04-12T22:06:06.468 INFO:teuthology.orchestra.run.smithi097.stdout:mgr.x smithi114 *:8443,9283,8765 running (3m) 19s ago 14m 451M - 19.0.0-2053-gb277feb2 668895944f0e b8f8df3bf928 2024-04-12T22:06:06.468 INFO:teuthology.orchestra.run.smithi097.stdout:mgr.y smithi097 *:8443,9283,8765 running (4m) 38s ago 16m 506M - 19.0.0-2053-gb277feb2 668895944f0e 488170dac11f 2024-04-12T22:06:06.468 INFO:teuthology.orchestra.run.smithi097.stdout:mon.a smithi097 running (3m) 38s ago 16m 50.6M 2048M 19.0.0-2053-gb277feb2 668895944f0e 3cfa16d3a85c 2024-04-12T22:06:06.468 INFO:teuthology.orchestra.run.smithi097.stdout:mon.b smithi114 running (2m) 19s ago 14m 36.8M 2048M 19.0.0-2053-gb277feb2 668895944f0e 71a2eff159a3 2024-04-12T22:06:06.468 INFO:teuthology.orchestra.run.smithi097.stdout:mon.c smithi097 running (3m) 38s ago 14m 39.3M 2048M 19.0.0-2053-gb277feb2 668895944f0e 69507ba88603 2024-04-12T22:06:06.468 INFO:teuthology.orchestra.run.smithi097.stdout:node-exporter.a smithi097 *:9100 running (5m) 38s ago 10m 17.7M - 1.3.1 1dbe0e931976 5edff9053b37 2024-04-12T22:06:06.468 INFO:teuthology.orchestra.run.smithi097.stdout:node-exporter.b smithi114 *:9100 running (4m) 19s ago 10m 17.1M - 1.3.1 1dbe0e931976 904afc45dc2f 2024-04-12T22:06:06.468 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0 smithi097 running (119s) 38s ago 14m 51.2M 2736M 19.0.0-2053-gb277feb2 668895944f0e 9d05fea4bf37 2024-04-12T22:06:06.468 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1 smithi097 running (99s) 38s ago 13m 43.7M 2736M 19.0.0-2053-gb277feb2 668895944f0e a9a527642889 2024-04-12T22:06:06.469 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2 smithi097 running (63s) 38s ago 13m 43.7M 2736M 19.0.0-2053-gb277feb2 668895944f0e 2e2297afff50 2024-04-12T22:06:06.469 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3 smithi097 running (43s) 38s ago 12m 12.6M 2736M 19.0.0-2053-gb277feb2 668895944f0e 415b4d3d85f2 2024-04-12T22:06:06.469 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4 smithi114 running (23s) 19s ago 12m 12.0M 2992M 19.0.0-2053-gb277feb2 668895944f0e fcd2e39a5866 2024-04-12T22:06:06.469 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5 smithi114 running (12m) 19s ago 12m 55.9M 2992M 17.2.0 e1d6a67b021e 3acd4a3b61e5 2024-04-12T22:06:06.469 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6 smithi114 running (11m) 19s ago 11m 54.0M 2992M 17.2.0 e1d6a67b021e 5969baf67f41 2024-04-12T22:06:06.469 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7 smithi114 running (11m) 19s ago 11m 57.2M 2992M 17.2.0 e1d6a67b021e f68340fecf2c 2024-04-12T22:06:06.469 INFO:teuthology.orchestra.run.smithi097.stdout:prometheus.a smithi114 *:9095 running (3m) 19s ago 10m 71.7M - 2.33.4 514e6a882f6e 0ad5437ed63f 2024-04-12T22:06:06.469 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.foo.smithi097.sggtkh smithi097 *:8000 running (9m) 38s ago 9m 73.0M - 17.2.0 e1d6a67b021e a39f9b2d8509 2024-04-12T22:06:06.469 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.foo.smithi114.ulfbyo smithi114 *:8000 running (9m) 19s ago 9m 73.4M - 17.2.0 e1d6a67b021e df79a48bdd59 2024-04-12T22:06:06.469 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.smpl.smithi097.exxkgl smithi097 *:80 running (9m) 38s ago 9m 72.9M - 17.2.0 e1d6a67b021e a036c58a0099 2024-04-12T22:06:06.469 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.smpl.smithi114.lrgvph smithi114 *:80 running (8m) 19s ago 8m 74.9M - 17.2.0 e1d6a67b021e 031a5fc24237 2024-04-12T22:06:06.861 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-12T22:06:06.861 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": { 2024-04-12T22:06:06.861 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 3 2024-04-12T22:06:06.861 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:06:06.861 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": { 2024-04-12T22:06:06.861 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 2 2024-04-12T22:06:06.861 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:06:06.861 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": { 2024-04-12T22:06:06.861 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-04-12T22:06:06.862 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 5 2024-04-12T22:06:06.862 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:06:06.862 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw": { 2024-04-12T22:06:06.862 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-12T22:06:06.862 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:06:06.862 INFO:teuthology.orchestra.run.smithi097.stdout: "overall": { 2024-04-12T22:06:06.862 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-04-12T22:06:06.862 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 10 2024-04-12T22:06:06.862 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-12T22:06:06.862 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-12T22:06:07.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:06 smithi097 ceph-mon[64853]: from='client.44308 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:06:07.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:06 smithi097 ceph-mon[67407]: from='client.44308 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:06:07.190 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-12T22:06:07.190 INFO:teuthology.orchestra.run.smithi097.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a", 2024-04-12T22:06:07.190 INFO:teuthology.orchestra.run.smithi097.stdout: "in_progress": true, 2024-04-12T22:06:07.190 INFO:teuthology.orchestra.run.smithi097.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T22:06:07.190 INFO:teuthology.orchestra.run.smithi097.stdout: "services_complete": [ 2024-04-12T22:06:07.190 INFO:teuthology.orchestra.run.smithi097.stdout: "mon", 2024-04-12T22:06:07.190 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr" 2024-04-12T22:06:07.190 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-04-12T22:06:07.190 INFO:teuthology.orchestra.run.smithi097.stdout: "progress": "10/23 daemons upgraded", 2024-04-12T22:06:07.190 INFO:teuthology.orchestra.run.smithi097.stdout: "message": "Currently upgrading osd daemons", 2024-04-12T22:06:07.190 INFO:teuthology.orchestra.run.smithi097.stdout: "is_paused": false 2024-04-12T22:06:07.190 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-12T22:06:07.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:06 smithi114 ceph-mon[58780]: from='client.44308 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:06:07.578 INFO:teuthology.orchestra.run.smithi097.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-12T22:06:07.578 INFO:teuthology.orchestra.run.smithi097.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-12T22:06:07.578 INFO:teuthology.orchestra.run.smithi097.stdout: daemon grafana.a on smithi114 is in error state 2024-04-12T22:06:07.900 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:07 smithi114 ceph-mon[58780]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:06:07.901 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:07 smithi114 ceph-mon[58780]: from='client.44314 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:06:07.901 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:07 smithi114 ceph-mon[58780]: from='client.34392 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:06:07.901 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:07 smithi114 ceph-mon[58780]: from='client.? 172.21.15.97:0/3450011882' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:06:07.901 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:07 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-12T22:06:07.901 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:07 smithi114 ceph-mon[58780]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-12T22:06:07.901 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:07 smithi114 ceph-mon[58780]: Upgrade: osd.5 is safe to restart 2024-04-12T22:06:07.901 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:07 smithi114 ceph-mon[58780]: from='client.44332 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:06:07.901 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:07 smithi114 ceph-mon[58780]: from='client.? 172.21.15.97:0/3966709015' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T22:06:08.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:07 smithi097 ceph-mon[67407]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:06:08.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:07 smithi097 ceph-mon[67407]: from='client.44314 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:06:08.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:07 smithi097 ceph-mon[67407]: from='client.34392 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:06:08.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:07 smithi097 ceph-mon[67407]: from='client.? 172.21.15.97:0/3450011882' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:06:08.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:07 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-12T22:06:08.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:07 smithi097 ceph-mon[67407]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-12T22:06:08.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:07 smithi097 ceph-mon[67407]: Upgrade: osd.5 is safe to restart 2024-04-12T22:06:08.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:07 smithi097 ceph-mon[67407]: from='client.44332 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:06:08.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:07 smithi097 ceph-mon[67407]: from='client.? 172.21.15.97:0/3966709015' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T22:06:08.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:07 smithi097 ceph-mon[64853]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:06:08.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:07 smithi097 ceph-mon[64853]: from='client.44314 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:06:08.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:07 smithi097 ceph-mon[64853]: from='client.34392 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:06:08.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:07 smithi097 ceph-mon[64853]: from='client.? 172.21.15.97:0/3450011882' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:06:08.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:07 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-12T22:06:08.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:07 smithi097 ceph-mon[64853]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-12T22:06:08.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:07 smithi097 ceph-mon[64853]: Upgrade: osd.5 is safe to restart 2024-04-12T22:06:08.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:07 smithi097 ceph-mon[64853]: from='client.44332 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:06:08.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:07 smithi097 ceph-mon[64853]: from='client.? 172.21.15.97:0/3966709015' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T22:06:08.743 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:06:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:06:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"smithi097:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.97\", device_class=\"ssd\", hostname=\"smithi097\", instance=\"172.21.15.114:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.97\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-12T22:06:09.660 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:06:09 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:06:09] "GET /metrics HTTP/1.1" 200 38454 "" "Prometheus/2.33.4" 2024-04-12T22:06:09.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:09 smithi114 ceph-mon[58780]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:06:09.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:09 smithi114 ceph-mon[58780]: Upgrade: Updating osd.5 2024-04-12T22:06:09.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:09 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:09.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:09 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-12T22:06:09.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:09 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:06:09.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:09 smithi114 ceph-mon[58780]: Deploying daemon osd.5 on smithi114 2024-04-12T22:06:09.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:09 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:06:10.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:09 smithi097 ceph-mon[67407]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:06:10.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:09 smithi097 ceph-mon[67407]: Upgrade: Updating osd.5 2024-04-12T22:06:10.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:09 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:10.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:09 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-12T22:06:10.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:09 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:06:10.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:09 smithi097 ceph-mon[67407]: Deploying daemon osd.5 on smithi114 2024-04-12T22:06:10.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:09 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:06:10.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:09 smithi097 ceph-mon[64853]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:06:10.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:09 smithi097 ceph-mon[64853]: Upgrade: Updating osd.5 2024-04-12T22:06:10.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:09 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:10.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:09 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-12T22:06:10.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:09 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:06:10.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:09 smithi097 ceph-mon[64853]: Deploying daemon osd.5 on smithi114 2024-04-12T22:06:10.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:09 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:06:11.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:11 smithi114 ceph-mon[58780]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:06:12.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:11 smithi097 ceph-mon[64853]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:06:12.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:11 smithi097 ceph-mon[67407]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:06:12.341 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:06:12 smithi114 systemd[1]: Stopping Ceph osd.5 for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:06:12.743 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:06:12 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5[32059]: 2024-04-12T22:06:12.339+0000 7f6cd8ebc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-12T22:06:12.743 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:06:12 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5[32059]: 2024-04-12T22:06:12.339+0000 7f6cd8ebc700 -1 osd.5 135 *** Got signal Terminated *** 2024-04-12T22:06:12.743 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:06:12 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5[32059]: 2024-04-12T22:06:12.339+0000 7f6cd8ebc700 -1 osd.5 135 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-12T22:06:13.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:12 smithi097 ceph-mon[64853]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:06:13.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:12 smithi097 ceph-mon[64853]: osd.5 marked itself down and dead 2024-04-12T22:06:13.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:12 smithi097 ceph-mon[67407]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:06:13.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:12 smithi097 ceph-mon[67407]: osd.5 marked itself down and dead 2024-04-12T22:06:13.240 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:12 smithi114 ceph-mon[58780]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:06:13.240 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:12 smithi114 ceph-mon[58780]: osd.5 marked itself down and dead 2024-04-12T22:06:13.493 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:06:13 smithi114 podman[64266]: 2024-04-12 22:06:13.239391896 +0000 UTC m=+1.042026603 container died 3acd4a3b61e59bfaf5d2e0d6f95fbca781f83a4ca0a5f685d2fca9a18cd1031c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, release=754, RELEASE=HEAD, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, version=8, distribution-scope=public, GIT_CLEAN=True) 2024-04-12T22:06:13.766 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:06:13 smithi114 podman[64266]: 2024-04-12 22:06:13.581689073 +0000 UTC m=+1.384323782 container cleanup 3acd4a3b61e59bfaf5d2e0d6f95fbca781f83a4ca0a5f685d2fca9a18cd1031c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, RELEASE=HEAD, maintainer=Guillaume Abrioux , name=centos-stream, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, release=754, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64) 2024-04-12T22:06:13.766 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:06:13 smithi114 bash[64266]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5 2024-04-12T22:06:14.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:13 smithi097 ceph-mon[64853]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:06:14.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:13 smithi097 ceph-mon[64853]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-12T22:06:14.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:13 smithi097 ceph-mon[64853]: osdmap e136: 8 total, 7 up, 8 in 2024-04-12T22:06:14.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:13 smithi097 ceph-mon[67407]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:06:14.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:13 smithi097 ceph-mon[67407]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-12T22:06:14.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:13 smithi097 ceph-mon[67407]: osdmap e136: 8 total, 7 up, 8 in 2024-04-12T22:06:14.207 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:13 smithi114 ceph-mon[58780]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:06:14.208 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:13 smithi114 ceph-mon[58780]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-12T22:06:14.208 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:13 smithi114 ceph-mon[58780]: osdmap e136: 8 total, 7 up, 8 in 2024-04-12T22:06:14.208 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:06:13 smithi114 podman[64283]: 2024-04-12 22:06:13.766080007 +0000 UTC m=+0.523020017 container remove 3acd4a3b61e59bfaf5d2e0d6f95fbca781f83a4ca0a5f685d2fca9a18cd1031c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, vcs-type=git, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., release=754, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, name=centos-stream) 2024-04-12T22:06:14.493 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:06:14 smithi114 podman[64377]: 2024-04-12 22:06:14.111008798 +0000 UTC m=+0.017942338 image pull 668895944f0ec64b67a5b7564961d2756f2966e5d4b9338516300dbec9984951 quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a 2024-04-12T22:06:14.493 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:06:14 smithi114 podman[64377]: 2024-04-12 22:06:14.208643046 +0000 UTC m=+0.115576572 container create 54c80706d9da5b114c67e119c2b85c704a82dc845c702d4ab9a071aa120e0fc8 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, RELEASE=squid-b277feb) 2024-04-12T22:06:14.493 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:06:14 smithi114 podman[64377]: 2024-04-12 22:06:14.419336068 +0000 UTC m=+0.326269594 container init 54c80706d9da5b114c67e119c2b85c704a82dc845c702d4ab9a071aa120e0fc8 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5-deactivate, RELEASE=squid-b277feb, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-12T22:06:14.493 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:06:14 smithi114 podman[64377]: 2024-04-12 22:06:14.422773095 +0000 UTC m=+0.329706632 container start 54c80706d9da5b114c67e119c2b85c704a82dc845c702d4ab9a071aa120e0fc8 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, RELEASE=squid-b277feb, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True) 2024-04-12T22:06:14.855 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:06:14 smithi114 podman[64377]: 2024-04-12 22:06:14.494370113 +0000 UTC m=+0.401303651 container attach 54c80706d9da5b114c67e119c2b85c704a82dc845c702d4ab9a071aa120e0fc8 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-b277feb, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-12T22:06:14.855 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:06:14 smithi114 podman[64377]: 2024-04-12 22:06:14.716738382 +0000 UTC m=+0.623671910 container died 54c80706d9da5b114c67e119c2b85c704a82dc845c702d4ab9a071aa120e0fc8 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, RELEASE=squid-b277feb, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-12T22:06:15.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:14 smithi097 ceph-mon[64853]: osdmap e137: 8 total, 7 up, 8 in 2024-04-12T22:06:15.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:14 smithi097 ceph-mon[67407]: osdmap e137: 8 total, 7 up, 8 in 2024-04-12T22:06:15.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:14 smithi114 ceph-mon[58780]: osdmap e137: 8 total, 7 up, 8 in 2024-04-12T22:06:15.716 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:06:15 smithi114 podman[64403]: 2024-04-12 22:06:15.289912219 +0000 UTC m=+0.568064580 container remove 54c80706d9da5b114c67e119c2b85c704a82dc845c702d4ab9a071aa120e0fc8 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-b277feb, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-12T22:06:15.716 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:06:15 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@osd.5.service: Deactivated successfully. 2024-04-12T22:06:15.716 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:06:15 smithi114 systemd[1]: Stopped Ceph osd.5 for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:06:15.716 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:06:15 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@osd.5.service: Consumed 8.189s CPU time. 2024-04-12T22:06:15.716 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:06:15 smithi114 systemd[1]: Starting Ceph osd.5 for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:06:15.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:15 smithi114 ceph-mon[58780]: pgmap v152: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:06:15.993 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:06:15 smithi114 podman[64512]: 2024-04-12 22:06:15.715766576 +0000 UTC m=+0.110063602 container create ab184449f965a4de521bc8ee06e4cdba3a61b6c970f7778653097c418f85d99d (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5-activate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, RELEASE=squid-b277feb, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-12T22:06:15.993 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:06:15 smithi114 podman[64512]: 2024-04-12 22:06:15.623871008 +0000 UTC m=+0.018168035 image pull 668895944f0ec64b67a5b7564961d2756f2966e5d4b9338516300dbec9984951 quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a 2024-04-12T22:06:15.993 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:06:15 smithi114 podman[64512]: 2024-04-12 22:06:15.960162821 +0000 UTC m=+0.354459839 container init ab184449f965a4de521bc8ee06e4cdba3a61b6c970f7778653097c418f85d99d (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5-activate, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, RELEASE=squid-b277feb, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-12T22:06:15.993 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:06:15 smithi114 podman[64512]: 2024-04-12 22:06:15.963466527 +0000 UTC m=+0.357763547 container start ab184449f965a4de521bc8ee06e4cdba3a61b6c970f7778653097c418f85d99d (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5-activate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, RELEASE=squid-b277feb, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-12T22:06:16.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:15 smithi097 ceph-mon[64853]: pgmap v152: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:06:16.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:15 smithi097 ceph-mon[67407]: pgmap v152: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:06:16.396 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:06:16 smithi114 podman[64512]: 2024-04-12 22:06:16.043564296 +0000 UTC m=+0.437861317 container attach ab184449f965a4de521bc8ee06e4cdba3a61b6c970f7778653097c418f85d99d (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5-activate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=squid-b277feb, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-12T22:06:16.743 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:06:16 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5-activate[64526]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-12T22:06:16.744 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:06:16 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5-activate[64526]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-12T22:06:16.744 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:06:16 smithi114 bash[64512]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-12T22:06:16.744 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:06:16 smithi114 bash[64512]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-12T22:06:16.744 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:06:16 smithi114 bash[64512]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-12T22:06:16.744 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:06:16 smithi114 bash[64512]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-12T22:06:16.744 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:06:16 smithi114 bash[64512]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-04-12T22:06:16.744 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:06:16 smithi114 bash[64512]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-12T22:06:16.744 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:06:16 smithi114 bash[64512]: --> ceph-volume raw activate successful for osd ID: 5 2024-04-12T22:06:16.744 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:06:16 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5-activate[64526]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-12T22:06:16.744 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:06:16 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5-activate[64526]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-12T22:06:16.744 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:06:16 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5-activate[64526]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-04-12T22:06:16.744 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:06:16 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5-activate[64526]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-12T22:06:16.744 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:06:16 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5-activate[64526]: --> ceph-volume raw activate successful for osd ID: 5 2024-04-12T22:06:16.745 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:06:16 smithi114 podman[64512]: 2024-04-12 22:06:16.420051683 +0000 UTC m=+0.814348694 container died ab184449f965a4de521bc8ee06e4cdba3a61b6c970f7778653097c418f85d99d (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5-activate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=squid-b277feb, GIT_CLEAN=True, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-12T22:06:17.147 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:16 smithi114 ceph-mon[58780]: pgmap v153: 161 pgs: 7 active+undersized, 8 active+undersized+degraded, 16 stale+active+clean, 130 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 20/630 objects degraded (3.175%) 2024-04-12T22:06:17.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:16 smithi097 ceph-mon[64853]: pgmap v153: 161 pgs: 7 active+undersized, 8 active+undersized+degraded, 16 stale+active+clean, 130 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 20/630 objects degraded (3.175%) 2024-04-12T22:06:17.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:16 smithi097 ceph-mon[67407]: pgmap v153: 161 pgs: 7 active+undersized, 8 active+undersized+degraded, 16 stale+active+clean, 130 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 20/630 objects degraded (3.175%) 2024-04-12T22:06:17.471 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:06:17 smithi114 podman[64642]: 2024-04-12 22:06:17.147868322 +0000 UTC m=+0.721145510 container remove ab184449f965a4de521bc8ee06e4cdba3a61b6c970f7778653097c418f85d99d (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5-activate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, RELEASE=squid-b277feb, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-12T22:06:17.471 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:06:17 smithi114 podman[64708]: 2024-04-12 22:06:17.271218625 +0000 UTC m=+0.018493575 image pull 668895944f0ec64b67a5b7564961d2756f2966e5d4b9338516300dbec9984951 quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a 2024-04-12T22:06:17.472 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:06:17 smithi114 podman[64708]: 2024-04-12 22:06:17.382129545 +0000 UTC m=+0.129404536 container create 470c0780e42894b52fc281912e97afaba929ee847bf4fceee6b0a971098806d2 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5, RELEASE=squid-b277feb, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240409, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True) 2024-04-12T22:06:17.863 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:06:17 smithi114 podman[64708]: 2024-04-12 22:06:17.576553178 +0000 UTC m=+0.323828127 container init 470c0780e42894b52fc281912e97afaba929ee847bf4fceee6b0a971098806d2 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5, RELEASE=squid-b277feb, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-12T22:06:17.863 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:06:17 smithi114 podman[64708]: 2024-04-12 22:06:17.57953602 +0000 UTC m=+0.326810971 container start 470c0780e42894b52fc281912e97afaba929ee847bf4fceee6b0a971098806d2 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-b277feb, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-12T22:06:17.864 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:06:17 smithi114 ceph-osd[64727]: -- 172.21.15.114:0/802611870 <== mon.2 v2:172.21.15.114:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d1d9f63680 con 0x55d1d9f2cc00 2024-04-12T22:06:17.864 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:06:17 smithi114 bash[64708]: 470c0780e42894b52fc281912e97afaba929ee847bf4fceee6b0a971098806d2 2024-04-12T22:06:17.864 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:06:17 smithi114 systemd[1]: Started Ceph osd.5 for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:06:18.136 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:17 smithi114 ceph-mon[58780]: Health check failed: Degraded data redundancy: 20/630 objects degraded (3.175%), 8 pgs degraded (PG_DEGRADED) 2024-04-12T22:06:18.136 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:17 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:18.136 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:17 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:18.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:17 smithi097 ceph-mon[64853]: Health check failed: Degraded data redundancy: 20/630 objects degraded (3.175%), 8 pgs degraded (PG_DEGRADED) 2024-04-12T22:06:18.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:17 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:18.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:17 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:18.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:17 smithi097 ceph-mon[67407]: Health check failed: Degraded data redundancy: 20/630 objects degraded (3.175%), 8 pgs degraded (PG_DEGRADED) 2024-04-12T22:06:18.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:17 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:18.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:17 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:18.493 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:06:18 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5[64723]: 2024-04-12T22:06:18.135+0000 7fa9837547c0 -1 Falling back to public interface 2024-04-12T22:06:19.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:18 smithi097 ceph-mon[64853]: pgmap v154: 161 pgs: 38 active+undersized, 22 active+undersized+degraded, 101 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 74/630 objects degraded (11.746%) 2024-04-12T22:06:19.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:18 smithi097 ceph-mon[67407]: pgmap v154: 161 pgs: 38 active+undersized, 22 active+undersized+degraded, 101 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 74/630 objects degraded (11.746%) 2024-04-12T22:06:19.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:18 smithi114 ceph-mon[58780]: pgmap v154: 161 pgs: 38 active+undersized, 22 active+undersized+degraded, 101 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 74/630 objects degraded (11.746%) 2024-04-12T22:06:19.660 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:06:19 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:06:19] "GET /metrics HTTP/1.1" 200 38453 "" "Prometheus/2.33.4" 2024-04-12T22:06:21.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:20 smithi097 ceph-mon[64853]: pgmap v155: 161 pgs: 38 active+undersized, 22 active+undersized+degraded, 101 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 74/630 objects degraded (11.746%) 2024-04-12T22:06:21.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:20 smithi097 ceph-mon[64853]: from='osd.5 [v2:172.21.15.114:6808/2352087710,v1:172.21.15.114:6809/2352087710]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-12T22:06:21.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:20 smithi097 ceph-mon[64853]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-12T22:06:21.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:20 smithi097 ceph-mon[67407]: pgmap v155: 161 pgs: 38 active+undersized, 22 active+undersized+degraded, 101 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 74/630 objects degraded (11.746%) 2024-04-12T22:06:21.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:20 smithi097 ceph-mon[67407]: from='osd.5 [v2:172.21.15.114:6808/2352087710,v1:172.21.15.114:6809/2352087710]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-12T22:06:21.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:20 smithi097 ceph-mon[67407]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-12T22:06:21.195 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:06:20 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5[64723]: 2024-04-12T22:06:20.856+0000 7fa9837547c0 -1 osd.5 0 read_superblock omap replica is missing. 2024-04-12T22:06:21.195 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:06:20 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5[64723]: 2024-04-12T22:06:20.882+0000 7fa9837547c0 -1 osd.5 135 log_to_monitors true 2024-04-12T22:06:21.195 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:20 smithi114 ceph-mon[58780]: pgmap v155: 161 pgs: 38 active+undersized, 22 active+undersized+degraded, 101 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 74/630 objects degraded (11.746%) 2024-04-12T22:06:21.195 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:20 smithi114 ceph-mon[58780]: from='osd.5 [v2:172.21.15.114:6808/2352087710,v1:172.21.15.114:6809/2352087710]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-12T22:06:21.195 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:20 smithi114 ceph-mon[58780]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-12T22:06:21.493 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:06:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5[64723]: 2024-04-12T22:06:21.422+0000 7fa970ba4700 -1 osd.5 135 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T22:06:22.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:21 smithi114 ceph-mon[58780]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:06:22.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:21 smithi114 ceph-mon[58780]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-12T22:06:22.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:21 smithi114 ceph-mon[58780]: from='osd.5 [v2:172.21.15.114:6808/2352087710,v1:172.21.15.114:6809/2352087710]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-12T22:06:22.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:21 smithi114 ceph-mon[58780]: osdmap e138: 8 total, 7 up, 8 in 2024-04-12T22:06:22.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:21 smithi114 ceph-mon[58780]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-12T22:06:22.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:21 smithi114 ceph-mon[58780]: from='osd.5 ' entity='osd.5' 2024-04-12T22:06:22.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:21 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:22.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:21 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:22.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:21 smithi097 ceph-mon[64853]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:06:22.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:21 smithi097 ceph-mon[64853]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-12T22:06:22.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:21 smithi097 ceph-mon[64853]: from='osd.5 [v2:172.21.15.114:6808/2352087710,v1:172.21.15.114:6809/2352087710]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-12T22:06:22.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:21 smithi097 ceph-mon[64853]: osdmap e138: 8 total, 7 up, 8 in 2024-04-12T22:06:22.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:21 smithi097 ceph-mon[64853]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-12T22:06:22.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:21 smithi097 ceph-mon[64853]: from='osd.5 ' entity='osd.5' 2024-04-12T22:06:22.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:21 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:22.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:21 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:22.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:21 smithi097 ceph-mon[67407]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:06:22.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:21 smithi097 ceph-mon[67407]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-12T22:06:22.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:21 smithi097 ceph-mon[67407]: from='osd.5 [v2:172.21.15.114:6808/2352087710,v1:172.21.15.114:6809/2352087710]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-12T22:06:22.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:21 smithi097 ceph-mon[67407]: osdmap e138: 8 total, 7 up, 8 in 2024-04-12T22:06:22.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:21 smithi097 ceph-mon[67407]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-12T22:06:22.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:21 smithi097 ceph-mon[67407]: from='osd.5 ' entity='osd.5' 2024-04-12T22:06:22.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:21 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:22.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:21 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:23.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:22 smithi114 ceph-mon[58780]: pgmap v157: 161 pgs: 38 active+undersized, 22 active+undersized+degraded, 101 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 74/630 objects degraded (11.746%) 2024-04-12T22:06:23.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:22 smithi114 ceph-mon[58780]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-12T22:06:23.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:22 smithi114 ceph-mon[58780]: osd.5 [v2:172.21.15.114:6808/2352087710,v1:172.21.15.114:6809/2352087710] boot 2024-04-12T22:06:23.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:22 smithi114 ceph-mon[58780]: osdmap e139: 8 total, 8 up, 8 in 2024-04-12T22:06:23.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:22 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T22:06:23.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:22 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:23.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:22 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:23.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:22 smithi097 ceph-mon[64853]: pgmap v157: 161 pgs: 38 active+undersized, 22 active+undersized+degraded, 101 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 74/630 objects degraded (11.746%) 2024-04-12T22:06:23.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:22 smithi097 ceph-mon[64853]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-12T22:06:23.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:22 smithi097 ceph-mon[64853]: osd.5 [v2:172.21.15.114:6808/2352087710,v1:172.21.15.114:6809/2352087710] boot 2024-04-12T22:06:23.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:22 smithi097 ceph-mon[64853]: osdmap e139: 8 total, 8 up, 8 in 2024-04-12T22:06:23.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:22 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T22:06:23.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:22 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:23.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:22 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:23.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:22 smithi097 ceph-mon[67407]: pgmap v157: 161 pgs: 38 active+undersized, 22 active+undersized+degraded, 101 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 74/630 objects degraded (11.746%) 2024-04-12T22:06:23.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:22 smithi097 ceph-mon[67407]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-12T22:06:23.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:22 smithi097 ceph-mon[67407]: osd.5 [v2:172.21.15.114:6808/2352087710,v1:172.21.15.114:6809/2352087710] boot 2024-04-12T22:06:23.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:22 smithi097 ceph-mon[67407]: osdmap e139: 8 total, 8 up, 8 in 2024-04-12T22:06:23.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:22 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T22:06:23.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:22 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:23.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:22 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:24.718 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:24 smithi114 ceph-mon[58780]: osdmap e140: 8 total, 8 up, 8 in 2024-04-12T22:06:24.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:24 smithi097 ceph-mon[64853]: osdmap e140: 8 total, 8 up, 8 in 2024-04-12T22:06:24.910 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:24 smithi097 ceph-mon[67407]: osdmap e140: 8 total, 8 up, 8 in 2024-04-12T22:06:26.115 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:25 smithi114 ceph-mon[58780]: pgmap v160: 161 pgs: 38 active+undersized, 22 active+undersized+degraded, 101 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 74/630 objects degraded (11.746%) 2024-04-12T22:06:26.115 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:25 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:26.115 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:25 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:06:26.115 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:25 smithi114 ceph-mon[58780]: Health check update: Degraded data redundancy: 74/630 objects degraded (11.746%), 22 pgs degraded (PG_DEGRADED) 2024-04-12T22:06:26.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:25 smithi097 ceph-mon[64853]: pgmap v160: 161 pgs: 38 active+undersized, 22 active+undersized+degraded, 101 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 74/630 objects degraded (11.746%) 2024-04-12T22:06:26.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:25 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:26.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:25 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:06:26.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:25 smithi097 ceph-mon[64853]: Health check update: Degraded data redundancy: 74/630 objects degraded (11.746%), 22 pgs degraded (PG_DEGRADED) 2024-04-12T22:06:26.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:25 smithi097 ceph-mon[67407]: pgmap v160: 161 pgs: 38 active+undersized, 22 active+undersized+degraded, 101 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 74/630 objects degraded (11.746%) 2024-04-12T22:06:26.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:25 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:26.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:25 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:06:26.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:25 smithi097 ceph-mon[67407]: Health check update: Degraded data redundancy: 74/630 objects degraded (11.746%), 22 pgs degraded (PG_DEGRADED) 2024-04-12T22:06:26.660 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:06:26 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:06:26.334+0000 7fe40fa5a700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (5 PGs are or would become offline) 2024-04-12T22:06:27.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:27 smithi114 ceph-mon[58780]: pgmap v161: 161 pgs: 34 active+undersized, 16 active+undersized+degraded, 111 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 59/630 objects degraded (9.365%) 2024-04-12T22:06:27.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:27 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:27.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:27 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:27.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:27 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:06:27.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:27 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:06:27.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:27 smithi114 ceph-mon[58780]: pgmap v162: 161 pgs: 18 active+undersized, 5 active+undersized+degraded, 138 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 978 B/s rd, 0 op/s; 18/630 objects degraded (2.857%) 2024-04-12T22:06:27.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:27 smithi114 ceph-mon[58780]: pgmap v163: 161 pgs: 18 active+undersized, 5 active+undersized+degraded, 138 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 18/630 objects degraded (2.857%) 2024-04-12T22:06:27.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:27 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:27.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:27 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:06:27.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:27 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:06:27.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:27 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:06:27.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:27 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:06:27.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:27 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-12T22:06:27.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:27 smithi114 ceph-mon[58780]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-12T22:06:27.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:27 smithi114 ceph-mon[58780]: Upgrade: unsafe to stop osd(s) at this time (5 PGs are or would become offline) 2024-04-12T22:06:27.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:27 smithi097 ceph-mon[67407]: pgmap v161: 161 pgs: 34 active+undersized, 16 active+undersized+degraded, 111 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 59/630 objects degraded (9.365%) 2024-04-12T22:06:27.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:27 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:27.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:27 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:27.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:27 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:06:27.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:27 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:06:27.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:27 smithi097 ceph-mon[67407]: pgmap v162: 161 pgs: 18 active+undersized, 5 active+undersized+degraded, 138 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 978 B/s rd, 0 op/s; 18/630 objects degraded (2.857%) 2024-04-12T22:06:27.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:27 smithi097 ceph-mon[67407]: pgmap v163: 161 pgs: 18 active+undersized, 5 active+undersized+degraded, 138 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 18/630 objects degraded (2.857%) 2024-04-12T22:06:27.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:27 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:27.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:27 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:06:27.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:27 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:06:27.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:27 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:06:27.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:27 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:06:27.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:27 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-12T22:06:27.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:27 smithi097 ceph-mon[67407]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-12T22:06:27.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:27 smithi097 ceph-mon[67407]: Upgrade: unsafe to stop osd(s) at this time (5 PGs are or would become offline) 2024-04-12T22:06:27.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:27 smithi097 ceph-mon[64853]: pgmap v161: 161 pgs: 34 active+undersized, 16 active+undersized+degraded, 111 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 59/630 objects degraded (9.365%) 2024-04-12T22:06:27.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:27 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:27.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:27 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:27.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:27 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:06:27.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:27 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:06:27.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:27 smithi097 ceph-mon[64853]: pgmap v162: 161 pgs: 18 active+undersized, 5 active+undersized+degraded, 138 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 978 B/s rd, 0 op/s; 18/630 objects degraded (2.857%) 2024-04-12T22:06:27.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:27 smithi097 ceph-mon[64853]: pgmap v163: 161 pgs: 18 active+undersized, 5 active+undersized+degraded, 138 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 18/630 objects degraded (2.857%) 2024-04-12T22:06:27.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:27 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:27.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:27 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:06:27.663 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:27 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:06:27.663 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:27 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:06:27.663 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:27 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:06:27.663 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:27 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-12T22:06:27.663 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:27 smithi097 ceph-mon[64853]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-12T22:06:27.663 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:27 smithi097 ceph-mon[64853]: Upgrade: unsafe to stop osd(s) at this time (5 PGs are or would become offline) 2024-04-12T22:06:28.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:28 smithi097 ceph-mon[64853]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 18/630 objects degraded (2.857%), 5 pgs degraded) 2024-04-12T22:06:28.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:28 smithi097 ceph-mon[67407]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 18/630 objects degraded (2.857%), 5 pgs degraded) 2024-04-12T22:06:28.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:28 smithi114 ceph-mon[58780]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 18/630 objects degraded (2.857%), 5 pgs degraded) 2024-04-12T22:06:29.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:29 smithi097 ceph-mon[64853]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 710 B/s rd, 0 op/s 2024-04-12T22:06:29.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:29 smithi097 ceph-mon[67407]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 710 B/s rd, 0 op/s 2024-04-12T22:06:29.661 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:06:29 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:06:29] "GET /metrics HTTP/1.1" 200 38322 "" "Prometheus/2.33.4" 2024-04-12T22:06:29.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:29 smithi114 ceph-mon[58780]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 710 B/s rd, 0 op/s 2024-04-12T22:06:31.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:31 smithi097 ceph-mon[64853]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 605 B/s rd, 0 op/s 2024-04-12T22:06:31.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:31 smithi097 ceph-mon[67407]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 605 B/s rd, 0 op/s 2024-04-12T22:06:31.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:31 smithi114 ceph-mon[58780]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 605 B/s rd, 0 op/s 2024-04-12T22:06:32.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:32 smithi097 ceph-mon[64853]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:06:32.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:32 smithi097 ceph-mon[67407]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:06:32.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:32 smithi114 ceph-mon[58780]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:06:33.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:33 smithi097 ceph-mon[64853]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-12T22:06:33.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:33 smithi097 ceph-mon[67407]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-12T22:06:33.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:33 smithi114 ceph-mon[58780]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-12T22:06:36.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:35 smithi097 ceph-mon[67407]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-12T22:06:36.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:35 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:36.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:35 smithi097 ceph-mon[64853]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-12T22:06:36.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:35 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:36.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:35 smithi114 ceph-mon[58780]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-12T22:06:36.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:35 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:37.938 INFO:teuthology.orchestra.run.smithi097.stdout:true 2024-04-12T22:06:38.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:37 smithi097 ceph-mon[64853]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-04-12T22:06:38.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:37 smithi097 ceph-mon[67407]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-04-12T22:06:38.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:37 smithi114 ceph-mon[58780]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-04-12T22:06:38.581 INFO:teuthology.orchestra.run.smithi097.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T22:06:38.581 INFO:teuthology.orchestra.run.smithi097.stdout:alertmanager.a smithi097 *:9093,9094 running (5m) 70s ago 10m 25.5M - 0.23.0 ba2b418f427c 7b4c491fe0a0 2024-04-12T22:06:38.581 INFO:teuthology.orchestra.run.smithi097.stdout:grafana.a smithi114 *:3000 error 16s ago 10m - - 2024-04-12T22:06:38.581 INFO:teuthology.orchestra.run.smithi097.stdout:iscsi.foo.smithi097.xndkxy smithi097 running (4m) 70s ago 9m 52.6M - 3.5 e1d6a67b021e 0436463f28cd 2024-04-12T22:06:38.581 INFO:teuthology.orchestra.run.smithi097.stdout:mgr.x smithi114 *:8443,9283,8765 running (4m) 16s ago 14m 451M - 19.0.0-2053-gb277feb2 668895944f0e b8f8df3bf928 2024-04-12T22:06:38.581 INFO:teuthology.orchestra.run.smithi097.stdout:mgr.y smithi097 *:8443,9283,8765 running (5m) 70s ago 16m 506M - 19.0.0-2053-gb277feb2 668895944f0e 488170dac11f 2024-04-12T22:06:38.581 INFO:teuthology.orchestra.run.smithi097.stdout:mon.a smithi097 running (4m) 70s ago 16m 50.6M 2048M 19.0.0-2053-gb277feb2 668895944f0e 3cfa16d3a85c 2024-04-12T22:06:38.581 INFO:teuthology.orchestra.run.smithi097.stdout:mon.b smithi114 running (3m) 16s ago 15m 38.6M 2048M 19.0.0-2053-gb277feb2 668895944f0e 71a2eff159a3 2024-04-12T22:06:38.581 INFO:teuthology.orchestra.run.smithi097.stdout:mon.c smithi097 running (3m) 70s ago 15m 39.3M 2048M 19.0.0-2053-gb277feb2 668895944f0e 69507ba88603 2024-04-12T22:06:38.582 INFO:teuthology.orchestra.run.smithi097.stdout:node-exporter.a smithi097 *:9100 running (5m) 70s ago 11m 17.7M - 1.3.1 1dbe0e931976 5edff9053b37 2024-04-12T22:06:38.582 INFO:teuthology.orchestra.run.smithi097.stdout:node-exporter.b smithi114 *:9100 running (5m) 16s ago 11m 17.2M - 1.3.1 1dbe0e931976 904afc45dc2f 2024-04-12T22:06:38.582 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0 smithi097 running (2m) 70s ago 14m 51.2M 2736M 19.0.0-2053-gb277feb2 668895944f0e 9d05fea4bf37 2024-04-12T22:06:38.582 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1 smithi097 running (2m) 70s ago 14m 43.7M 2736M 19.0.0-2053-gb277feb2 668895944f0e a9a527642889 2024-04-12T22:06:38.582 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2 smithi097 running (96s) 70s ago 13m 43.7M 2736M 19.0.0-2053-gb277feb2 668895944f0e 2e2297afff50 2024-04-12T22:06:38.582 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3 smithi097 running (75s) 70s ago 13m 12.6M 2736M 19.0.0-2053-gb277feb2 668895944f0e 415b4d3d85f2 2024-04-12T22:06:38.582 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4 smithi114 running (55s) 16s ago 13m 44.7M 2992M 19.0.0-2053-gb277feb2 668895944f0e fcd2e39a5866 2024-04-12T22:06:38.582 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5 smithi114 running (21s) 16s ago 12m 12.7M 2992M 19.0.0-2053-gb277feb2 668895944f0e 470c0780e428 2024-04-12T22:06:38.582 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6 smithi114 running (12m) 16s ago 12m 56.0M 2992M 17.2.0 e1d6a67b021e 5969baf67f41 2024-04-12T22:06:38.582 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7 smithi114 running (11m) 16s ago 11m 58.3M 2992M 17.2.0 e1d6a67b021e f68340fecf2c 2024-04-12T22:06:38.582 INFO:teuthology.orchestra.run.smithi097.stdout:prometheus.a smithi114 *:9095 running (4m) 16s ago 11m 71.8M - 2.33.4 514e6a882f6e 0ad5437ed63f 2024-04-12T22:06:38.582 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.foo.smithi097.sggtkh smithi097 *:8000 running (9m) 70s ago 9m 73.0M - 17.2.0 e1d6a67b021e a39f9b2d8509 2024-04-12T22:06:38.582 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.foo.smithi114.ulfbyo smithi114 *:8000 running (9m) 16s ago 9m 73.5M - 17.2.0 e1d6a67b021e df79a48bdd59 2024-04-12T22:06:38.582 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.smpl.smithi097.exxkgl smithi097 *:80 running (9m) 70s ago 9m 72.9M - 17.2.0 e1d6a67b021e a036c58a0099 2024-04-12T22:06:38.583 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.smpl.smithi114.lrgvph smithi114 *:80 running (9m) 16s ago 9m 75.1M - 17.2.0 e1d6a67b021e 031a5fc24237 2024-04-12T22:06:38.972 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-12T22:06:38.972 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": { 2024-04-12T22:06:38.972 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 3 2024-04-12T22:06:38.973 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:06:38.973 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": { 2024-04-12T22:06:38.973 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 2 2024-04-12T22:06:38.973 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:06:38.973 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": { 2024-04-12T22:06:38.973 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-04-12T22:06:38.973 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 6 2024-04-12T22:06:38.973 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:06:38.973 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw": { 2024-04-12T22:06:38.973 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-12T22:06:38.973 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:06:38.973 INFO:teuthology.orchestra.run.smithi097.stdout: "overall": { 2024-04-12T22:06:38.973 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-04-12T22:06:38.973 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 11 2024-04-12T22:06:38.974 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-12T22:06:38.974 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-12T22:06:39.300 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-12T22:06:39.300 INFO:teuthology.orchestra.run.smithi097.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a", 2024-04-12T22:06:39.300 INFO:teuthology.orchestra.run.smithi097.stdout: "in_progress": true, 2024-04-12T22:06:39.300 INFO:teuthology.orchestra.run.smithi097.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T22:06:39.300 INFO:teuthology.orchestra.run.smithi097.stdout: "services_complete": [ 2024-04-12T22:06:39.300 INFO:teuthology.orchestra.run.smithi097.stdout: "mon", 2024-04-12T22:06:39.300 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr" 2024-04-12T22:06:39.301 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-04-12T22:06:39.301 INFO:teuthology.orchestra.run.smithi097.stdout: "progress": "11/23 daemons upgraded", 2024-04-12T22:06:39.301 INFO:teuthology.orchestra.run.smithi097.stdout: "message": "Currently upgrading osd daemons", 2024-04-12T22:06:39.301 INFO:teuthology.orchestra.run.smithi097.stdout: "is_paused": false 2024-04-12T22:06:39.301 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-12T22:06:39.301 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:06:39 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:06:39] "GET /metrics HTTP/1.1" 200 38322 "" "Prometheus/2.33.4" 2024-04-12T22:06:39.695 INFO:teuthology.orchestra.run.smithi097.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-12T22:06:39.695 INFO:teuthology.orchestra.run.smithi097.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-12T22:06:39.696 INFO:teuthology.orchestra.run.smithi097.stdout: daemon grafana.a on smithi114 is in error state 2024-04-12T22:06:40.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:39 smithi097 ceph-mon[64853]: from='client.34422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:06:40.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:39 smithi097 ceph-mon[64853]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:06:40.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:39 smithi097 ceph-mon[64853]: from='client.44356 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:06:40.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:39 smithi097 ceph-mon[64853]: from='client.44362 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:06:40.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:39 smithi097 ceph-mon[64853]: from='client.? 172.21.15.97:0/928755155' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:06:40.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:39 smithi097 ceph-mon[64853]: from='client.? 172.21.15.97:0/3837008208' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T22:06:40.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:39 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:40.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:39 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:06:40.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:39 smithi097 ceph-mon[67407]: from='client.34422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:06:40.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:39 smithi097 ceph-mon[67407]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:06:40.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:39 smithi097 ceph-mon[67407]: from='client.44356 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:06:40.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:39 smithi097 ceph-mon[67407]: from='client.44362 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:06:40.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:39 smithi097 ceph-mon[67407]: from='client.? 172.21.15.97:0/928755155' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:06:40.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:39 smithi097 ceph-mon[67407]: from='client.? 172.21.15.97:0/3837008208' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T22:06:40.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:39 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:40.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:39 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:06:40.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:39 smithi114 ceph-mon[58780]: from='client.34422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:06:40.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:39 smithi114 ceph-mon[58780]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:06:40.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:39 smithi114 ceph-mon[58780]: from='client.44356 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:06:40.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:39 smithi114 ceph-mon[58780]: from='client.44362 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:06:40.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:39 smithi114 ceph-mon[58780]: from='client.? 172.21.15.97:0/928755155' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:06:40.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:39 smithi114 ceph-mon[58780]: from='client.? 172.21.15.97:0/3837008208' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T22:06:40.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:39 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:40.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:39 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:06:41.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:40 smithi097 ceph-mon[64853]: from='client.44374 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:06:41.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:40 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.0"}]: dispatch 2024-04-12T22:06:41.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:40 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.a"}]: dispatch 2024-04-12T22:06:41.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:40 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.12"}]: dispatch 2024-04-12T22:06:41.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:40 smithi097 ceph-mon[67407]: from='client.44374 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:06:41.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:40 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.0"}]: dispatch 2024-04-12T22:06:41.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:40 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.a"}]: dispatch 2024-04-12T22:06:41.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:40 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.12"}]: dispatch 2024-04-12T22:06:41.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:40 smithi114 ceph-mon[58780]: from='client.44374 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:06:41.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:40 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.0"}]: dispatch 2024-04-12T22:06:41.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:40 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.a"}]: dispatch 2024-04-12T22:06:41.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:40 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.12"}]: dispatch 2024-04-12T22:06:42.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:41 smithi097 ceph-mon[64853]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:06:42.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:41 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.0"}]': finished 2024-04-12T22:06:42.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:41 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.a"}]': finished 2024-04-12T22:06:42.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:41 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.12"}]': finished 2024-04-12T22:06:42.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:41 smithi097 ceph-mon[64853]: osdmap e141: 8 total, 8 up, 8 in 2024-04-12T22:06:42.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:41 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-12T22:06:42.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:41 smithi097 ceph-mon[67407]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:06:42.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:41 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.0"}]': finished 2024-04-12T22:06:42.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:41 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.a"}]': finished 2024-04-12T22:06:42.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:41 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.12"}]': finished 2024-04-12T22:06:42.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:41 smithi097 ceph-mon[67407]: osdmap e141: 8 total, 8 up, 8 in 2024-04-12T22:06:42.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:41 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-12T22:06:42.218 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:41 smithi114 ceph-mon[58780]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:06:42.218 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:41 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.0"}]': finished 2024-04-12T22:06:42.218 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:41 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.a"}]': finished 2024-04-12T22:06:42.218 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:41 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.12"}]': finished 2024-04-12T22:06:42.218 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:41 smithi114 ceph-mon[58780]: osdmap e141: 8 total, 8 up, 8 in 2024-04-12T22:06:42.218 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:41 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-12T22:06:43.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:42 smithi097 ceph-mon[64853]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:06:43.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:42 smithi097 ceph-mon[64853]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-12T22:06:43.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:42 smithi097 ceph-mon[64853]: Upgrade: osd.6 is safe to restart 2024-04-12T22:06:43.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:42 smithi097 ceph-mon[64853]: osdmap e142: 8 total, 8 up, 8 in 2024-04-12T22:06:43.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:42 smithi097 ceph-mon[64853]: pgmap v173: 161 pgs: 1 peering, 160 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:06:43.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:42 smithi097 ceph-mon[67407]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:06:43.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:42 smithi097 ceph-mon[67407]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-12T22:06:43.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:42 smithi097 ceph-mon[67407]: Upgrade: osd.6 is safe to restart 2024-04-12T22:06:43.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:42 smithi097 ceph-mon[67407]: osdmap e142: 8 total, 8 up, 8 in 2024-04-12T22:06:43.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:42 smithi097 ceph-mon[67407]: pgmap v173: 161 pgs: 1 peering, 160 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:06:43.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:42 smithi114 ceph-mon[58780]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:06:43.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:42 smithi114 ceph-mon[58780]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-12T22:06:43.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:42 smithi114 ceph-mon[58780]: Upgrade: osd.6 is safe to restart 2024-04-12T22:06:43.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:42 smithi114 ceph-mon[58780]: osdmap e142: 8 total, 8 up, 8 in 2024-04-12T22:06:43.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:42 smithi114 ceph-mon[58780]: pgmap v173: 161 pgs: 1 peering, 160 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:06:44.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:43 smithi097 ceph-mon[64853]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-12T22:06:44.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:43 smithi097 ceph-mon[64853]: Upgrade: Updating osd.6 2024-04-12T22:06:44.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:43 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:44.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:43 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-12T22:06:44.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:43 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:06:44.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:43 smithi097 ceph-mon[64853]: Deploying daemon osd.6 on smithi114 2024-04-12T22:06:44.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:43 smithi097 ceph-mon[67407]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-12T22:06:44.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:43 smithi097 ceph-mon[67407]: Upgrade: Updating osd.6 2024-04-12T22:06:44.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:43 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:44.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:43 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-12T22:06:44.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:43 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:06:44.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:43 smithi097 ceph-mon[67407]: Deploying daemon osd.6 on smithi114 2024-04-12T22:06:44.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:43 smithi114 ceph-mon[58780]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-12T22:06:44.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:43 smithi114 ceph-mon[58780]: Upgrade: Updating osd.6 2024-04-12T22:06:44.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:43 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:44.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:43 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-12T22:06:44.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:43 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:06:44.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:43 smithi114 ceph-mon[58780]: Deploying daemon osd.6 on smithi114 2024-04-12T22:06:45.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:44 smithi097 ceph-mon[64853]: pgmap v174: 161 pgs: 1 peering, 160 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:06:45.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:44 smithi097 ceph-mon[67407]: pgmap v174: 161 pgs: 1 peering, 160 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:06:45.220 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:44 smithi114 ceph-mon[58780]: pgmap v174: 161 pgs: 1 peering, 160 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:06:46.616 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:06:46 smithi114 systemd[1]: Stopping Ceph osd.6 for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:06:46.993 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:06:46 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[34949]: 2024-04-12T22:06:46.615+0000 7ff550570700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-12T22:06:46.993 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:06:46 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[34949]: 2024-04-12T22:06:46.615+0000 7ff550570700 -1 osd.6 142 *** Got signal Terminated *** 2024-04-12T22:06:46.993 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:06:46 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[34949]: 2024-04-12T22:06:46.615+0000 7ff550570700 -1 osd.6 142 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-12T22:06:47.588 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:47 smithi114 ceph-mon[58780]: pgmap v175: 161 pgs: 1 peering, 160 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-12T22:06:47.588 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:47 smithi114 ceph-mon[58780]: osd.6 marked itself down and dead 2024-04-12T22:06:47.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:47 smithi097 ceph-mon[64853]: pgmap v175: 161 pgs: 1 peering, 160 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-12T22:06:47.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:47 smithi097 ceph-mon[64853]: osd.6 marked itself down and dead 2024-04-12T22:06:47.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:47 smithi097 ceph-mon[67407]: pgmap v175: 161 pgs: 1 peering, 160 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-12T22:06:47.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:47 smithi097 ceph-mon[67407]: osd.6 marked itself down and dead 2024-04-12T22:06:47.892 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:06:47 smithi114 podman[66567]: 2024-04-12 22:06:47.587153012 +0000 UTC m=+1.084457620 container died 5969baf67f4192175a0094898964683a06ba3ec4ee35fa8ad7fba965285564a3 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, version=8, io.openshift.expose-services=, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, ceph=True, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux ) 2024-04-12T22:06:48.243 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:06:48 smithi114 podman[66567]: 2024-04-12 22:06:48.057590511 +0000 UTC m=+1.554895165 container cleanup 5969baf67f4192175a0094898964683a06ba3ec4ee35fa8ad7fba965285564a3 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, GIT_BRANCH=HEAD, version=8, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, vcs-type=git, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , name=centos-stream, com.redhat.component=centos-stream-container, architecture=x86_64, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-04-12T22:06:48.243 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:06:48 smithi114 bash[66567]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6 2024-04-12T22:06:48.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:48 smithi097 ceph-mon[64853]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-12T22:06:48.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:48 smithi097 ceph-mon[64853]: osdmap e143: 8 total, 7 up, 8 in 2024-04-12T22:06:48.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:48 smithi097 ceph-mon[67407]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-12T22:06:48.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:48 smithi097 ceph-mon[67407]: osdmap e143: 8 total, 7 up, 8 in 2024-04-12T22:06:48.718 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:48 smithi114 ceph-mon[58780]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-12T22:06:48.719 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:48 smithi114 ceph-mon[58780]: osdmap e143: 8 total, 7 up, 8 in 2024-04-12T22:06:48.719 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:06:48 smithi114 podman[66583]: 2024-04-12 22:06:48.392629155 +0000 UTC m=+0.801100973 container remove 5969baf67f4192175a0094898964683a06ba3ec4ee35fa8ad7fba965285564a3 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, version=8, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, distribution-scope=public, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, io.openshift.expose-services=, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream) 2024-04-12T22:06:48.972 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:06:48 smithi114 podman[66674]: 2024-04-12 22:06:48.718585922 +0000 UTC m=+0.113113755 container create a7b2ae3d1ac80b58585560f748a64f60079c2dbe5c34604bfba540f252fd9c2d (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-b277feb, GIT_BRANCH=HEAD) 2024-04-12T22:06:48.972 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:06:48 smithi114 podman[66674]: 2024-04-12 22:06:48.622910084 +0000 UTC m=+0.017437872 image pull 668895944f0ec64b67a5b7564961d2756f2966e5d4b9338516300dbec9984951 quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a 2024-04-12T22:06:49.243 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:06:48 smithi114 podman[66674]: 2024-04-12 22:06:48.971984733 +0000 UTC m=+0.366512516 container init a7b2ae3d1ac80b58585560f748a64f60079c2dbe5c34604bfba540f252fd9c2d (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b277feb, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-12T22:06:49.243 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:06:48 smithi114 podman[66674]: 2024-04-12 22:06:48.975459802 +0000 UTC m=+0.369987580 container start a7b2ae3d1ac80b58585560f748a64f60079c2dbe5c34604bfba540f252fd9c2d (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=squid-b277feb, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-12T22:06:49.243 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:06:49 smithi114 podman[66674]: 2024-04-12 22:06:49.088200722 +0000 UTC m=+0.482728497 container attach a7b2ae3d1ac80b58585560f748a64f60079c2dbe5c34604bfba540f252fd9c2d (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-b277feb, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-12T22:06:49.558 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:06:49 smithi114 podman[66674]: 2024-04-12 22:06:49.265300726 +0000 UTC m=+0.659828491 container died a7b2ae3d1ac80b58585560f748a64f60079c2dbe5c34604bfba540f252fd9c2d (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6-deactivate, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b277feb, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-12T22:06:49.558 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:49 smithi114 ceph-mon[58780]: pgmap v177: 161 pgs: 16 peering, 7 stale+active+clean, 138 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 279 B/s rd, 0 op/s 2024-04-12T22:06:49.558 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:49 smithi114 ceph-mon[58780]: Health check update: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-04-12T22:06:49.558 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:49 smithi114 ceph-mon[58780]: osdmap e144: 8 total, 7 up, 8 in 2024-04-12T22:06:49.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:49 smithi097 ceph-mon[64853]: pgmap v177: 161 pgs: 16 peering, 7 stale+active+clean, 138 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 279 B/s rd, 0 op/s 2024-04-12T22:06:49.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:49 smithi097 ceph-mon[64853]: Health check update: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-04-12T22:06:49.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:49 smithi097 ceph-mon[64853]: osdmap e144: 8 total, 7 up, 8 in 2024-04-12T22:06:49.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:49 smithi097 ceph-mon[67407]: pgmap v177: 161 pgs: 16 peering, 7 stale+active+clean, 138 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 279 B/s rd, 0 op/s 2024-04-12T22:06:49.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:49 smithi097 ceph-mon[67407]: Health check update: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-04-12T22:06:49.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:49 smithi097 ceph-mon[67407]: osdmap e144: 8 total, 7 up, 8 in 2024-04-12T22:06:49.661 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:06:49 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:06:49] "GET /metrics HTTP/1.1" 200 38469 "" "Prometheus/2.33.4" 2024-04-12T22:06:50.243 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:06:49 smithi114 podman[66700]: 2024-04-12 22:06:49.883646005 +0000 UTC m=+0.611993290 container remove a7b2ae3d1ac80b58585560f748a64f60079c2dbe5c34604bfba540f252fd9c2d (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-b277feb, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2) 2024-04-12T22:06:50.243 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:06:49 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@osd.6.service: Deactivated successfully. 2024-04-12T22:06:50.243 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:06:49 smithi114 systemd[1]: Stopped Ceph osd.6 for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:06:50.243 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:06:49 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@osd.6.service: Consumed 7.598s CPU time. 2024-04-12T22:06:50.243 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:06:50 smithi114 systemd[1]: Starting Ceph osd.6 for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:06:50.570 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:06:50 smithi114 podman[66801]: 2024-04-12 22:06:50.221125987 +0000 UTC m=+0.017770905 image pull 668895944f0ec64b67a5b7564961d2756f2966e5d4b9338516300dbec9984951 quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a 2024-04-12T22:06:50.570 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:06:50 smithi114 podman[66801]: 2024-04-12 22:06:50.334713158 +0000 UTC m=+0.131358062 container create c5fdbfaa31ca0d5f9f3e6e35986b4cd050e008159b179588630bf22acf3901b9 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6-activate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, RELEASE=squid-b277feb, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240409) 2024-04-12T22:06:50.993 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:06:50 smithi114 podman[66801]: 2024-04-12 22:06:50.570577686 +0000 UTC m=+0.367222591 container init c5fdbfaa31ca0d5f9f3e6e35986b4cd050e008159b179588630bf22acf3901b9 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6-activate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240409, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-b277feb, io.buildah.version=1.33.5) 2024-04-12T22:06:50.993 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:06:50 smithi114 podman[66801]: 2024-04-12 22:06:50.573611546 +0000 UTC m=+0.370256461 container start c5fdbfaa31ca0d5f9f3e6e35986b4cd050e008159b179588630bf22acf3901b9 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6-activate, org.label-schema.license=GPLv2, RELEASE=squid-b277feb, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0) 2024-04-12T22:06:50.993 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:06:50 smithi114 podman[66801]: 2024-04-12 22:06:50.678967825 +0000 UTC m=+0.475612741 container attach c5fdbfaa31ca0d5f9f3e6e35986b4cd050e008159b179588630bf22acf3901b9 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6-activate, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-b277feb, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-12T22:06:51.375 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:06:51 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6-activate[66816]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-12T22:06:51.375 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:06:51 smithi114 bash[66801]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-12T22:06:51.375 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:06:51 smithi114 bash[66801]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-12T22:06:51.375 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:06:51 smithi114 bash[66801]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-12T22:06:51.375 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:06:51 smithi114 bash[66801]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-12T22:06:51.375 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:06:51 smithi114 bash[66801]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-04-12T22:06:51.375 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:06:51 smithi114 bash[66801]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-12T22:06:51.376 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:06:51 smithi114 bash[66801]: --> ceph-volume raw activate successful for osd ID: 6 2024-04-12T22:06:51.376 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:06:51 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6-activate[66816]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-12T22:06:51.376 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:06:51 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6-activate[66816]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-12T22:06:51.376 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:06:51 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6-activate[66816]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-12T22:06:51.376 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:06:51 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6-activate[66816]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-04-12T22:06:51.376 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:06:51 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6-activate[66816]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-12T22:06:51.376 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:06:51 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6-activate[66816]: --> ceph-volume raw activate successful for osd ID: 6 2024-04-12T22:06:51.376 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:06:51 smithi114 podman[66801]: 2024-04-12 22:06:51.033647313 +0000 UTC m=+0.830292218 container died c5fdbfaa31ca0d5f9f3e6e35986b4cd050e008159b179588630bf22acf3901b9 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6-activate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-b277feb) 2024-04-12T22:06:51.376 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:51 smithi114 ceph-mon[58780]: pgmap v179: 161 pgs: 16 peering, 7 stale+active+clean, 138 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-04-12T22:06:51.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:51 smithi097 ceph-mon[64853]: pgmap v179: 161 pgs: 16 peering, 7 stale+active+clean, 138 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-04-12T22:06:51.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:51 smithi097 ceph-mon[67407]: pgmap v179: 161 pgs: 16 peering, 7 stale+active+clean, 138 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-04-12T22:06:51.993 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:06:51 smithi114 podman[66943]: 2024-04-12 22:06:51.674340367 +0000 UTC m=+0.635748492 container remove c5fdbfaa31ca0d5f9f3e6e35986b4cd050e008159b179588630bf22acf3901b9 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6-activate, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-b277feb, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-12T22:06:51.993 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:06:51 smithi114 podman[67007]: 2024-04-12 22:06:51.790472022 +0000 UTC m=+0.017544484 image pull 668895944f0ec64b67a5b7564961d2756f2966e5d4b9338516300dbec9984951 quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a 2024-04-12T22:06:51.993 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:06:51 smithi114 podman[67007]: 2024-04-12 22:06:51.911895255 +0000 UTC m=+0.138967693 container create 6b80aaeca08bb3819b8a392239756b62b64b2f5a3e753af4da236521d31ee897 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=squid-b277feb, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0) 2024-04-12T22:06:52.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:52 smithi114 ceph-mon[58780]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:06:52.493 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:06:52 smithi114 podman[67007]: 2024-04-12 22:06:52.169862276 +0000 UTC m=+0.396934718 container init 6b80aaeca08bb3819b8a392239756b62b64b2f5a3e753af4da236521d31ee897 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-b277feb, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409) 2024-04-12T22:06:52.493 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:06:52 smithi114 podman[67007]: 2024-04-12 22:06:52.173333079 +0000 UTC m=+0.400405519 container start 6b80aaeca08bb3819b8a392239756b62b64b2f5a3e753af4da236521d31ee897 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-b277feb) 2024-04-12T22:06:52.493 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:06:52 smithi114 ceph-osd[67027]: -- 172.21.15.114:0/3262337225 <== mon.2 v2:172.21.15.114:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55bcb4d53680 con 0x55bcb4d1d400 2024-04-12T22:06:52.493 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:06:52 smithi114 bash[67007]: 6b80aaeca08bb3819b8a392239756b62b64b2f5a3e753af4da236521d31ee897 2024-04-12T22:06:52.493 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:06:52 smithi114 systemd[1]: Started Ceph osd.6 for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:06:52.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:52 smithi097 ceph-mon[64853]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:06:52.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:52 smithi097 ceph-mon[67407]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:06:52.993 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:06:52 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:06:52.718+0000 7fc7b3bf77c0 -1 Falling back to public interface 2024-04-12T22:06:53.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:53 smithi114 ceph-mon[58780]: pgmap v180: 161 pgs: 24 active+undersized, 13 active+undersized+degraded, 16 peering, 108 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 51/630 objects degraded (8.095%); 0 B/s, 0 objects/s recovering 2024-04-12T22:06:53.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:53 smithi114 ceph-mon[58780]: Health check failed: Degraded data redundancy: 51/630 objects degraded (8.095%), 13 pgs degraded (PG_DEGRADED) 2024-04-12T22:06:53.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:53 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:53.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:53 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:53.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:53 smithi097 ceph-mon[67407]: pgmap v180: 161 pgs: 24 active+undersized, 13 active+undersized+degraded, 16 peering, 108 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 51/630 objects degraded (8.095%); 0 B/s, 0 objects/s recovering 2024-04-12T22:06:53.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:53 smithi097 ceph-mon[67407]: Health check failed: Degraded data redundancy: 51/630 objects degraded (8.095%), 13 pgs degraded (PG_DEGRADED) 2024-04-12T22:06:53.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:53 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:53.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:53 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:53.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:53 smithi097 ceph-mon[64853]: pgmap v180: 161 pgs: 24 active+undersized, 13 active+undersized+degraded, 16 peering, 108 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 51/630 objects degraded (8.095%); 0 B/s, 0 objects/s recovering 2024-04-12T22:06:53.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:53 smithi097 ceph-mon[64853]: Health check failed: Degraded data redundancy: 51/630 objects degraded (8.095%), 13 pgs degraded (PG_DEGRADED) 2024-04-12T22:06:53.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:53 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:53.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:53 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:54.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:54 smithi097 ceph-mon[64853]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 6 pgs peering) 2024-04-12T22:06:54.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:54 smithi097 ceph-mon[67407]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 6 pgs peering) 2024-04-12T22:06:54.694 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:54 smithi114 ceph-mon[58780]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 6 pgs peering) 2024-04-12T22:06:55.743 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:06:55 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:06:55.428+0000 7fc7b3bf77c0 -1 osd.6 0 read_superblock omap replica is missing. 2024-04-12T22:06:55.743 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:06:55 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:06:55.455+0000 7fc7b3bf77c0 -1 osd.6 142 log_to_monitors true 2024-04-12T22:06:56.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:55 smithi097 ceph-mon[64853]: pgmap v181: 161 pgs: 34 active+undersized, 19 active+undersized+degraded, 108 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 76/630 objects degraded (12.063%); 0 B/s, 0 objects/s recovering 2024-04-12T22:06:56.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:55 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:56.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:55 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:06:56.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:55 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:56.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:55 smithi097 ceph-mon[64853]: from='osd.6 [v2:172.21.15.114:6816/4257502981,v1:172.21.15.114:6817/4257502981]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-12T22:06:56.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:55 smithi097 ceph-mon[64853]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-12T22:06:56.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:55 smithi097 ceph-mon[67407]: pgmap v181: 161 pgs: 34 active+undersized, 19 active+undersized+degraded, 108 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 76/630 objects degraded (12.063%); 0 B/s, 0 objects/s recovering 2024-04-12T22:06:56.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:55 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:56.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:55 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:06:56.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:55 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:56.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:55 smithi097 ceph-mon[67407]: from='osd.6 [v2:172.21.15.114:6816/4257502981,v1:172.21.15.114:6817/4257502981]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-12T22:06:56.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:55 smithi097 ceph-mon[67407]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-12T22:06:56.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:55 smithi114 ceph-mon[58780]: pgmap v181: 161 pgs: 34 active+undersized, 19 active+undersized+degraded, 108 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 76/630 objects degraded (12.063%); 0 B/s, 0 objects/s recovering 2024-04-12T22:06:56.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:55 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:56.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:55 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:06:56.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:55 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:56.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:55 smithi114 ceph-mon[58780]: from='osd.6 [v2:172.21.15.114:6816/4257502981,v1:172.21.15.114:6817/4257502981]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-12T22:06:56.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:55 smithi114 ceph-mon[58780]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-12T22:06:56.743 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:06:56 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:06:56.353+0000 7fc7a1047700 -1 osd.6 142 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T22:06:57.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:56 smithi097 ceph-mon[64853]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-12T22:06:57.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:56 smithi097 ceph-mon[64853]: from='osd.6 [v2:172.21.15.114:6816/4257502981,v1:172.21.15.114:6817/4257502981]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-12T22:06:57.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:56 smithi097 ceph-mon[64853]: osdmap e145: 8 total, 7 up, 8 in 2024-04-12T22:06:57.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:56 smithi097 ceph-mon[64853]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-12T22:06:57.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:56 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:57.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:56 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:57.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:56 smithi097 ceph-mon[64853]: from='osd.6 ' entity='osd.6' 2024-04-12T22:06:57.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:56 smithi097 ceph-mon[67407]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-12T22:06:57.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:56 smithi097 ceph-mon[67407]: from='osd.6 [v2:172.21.15.114:6816/4257502981,v1:172.21.15.114:6817/4257502981]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-12T22:06:57.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:56 smithi097 ceph-mon[67407]: osdmap e145: 8 total, 7 up, 8 in 2024-04-12T22:06:57.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:56 smithi097 ceph-mon[67407]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-12T22:06:57.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:56 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:57.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:56 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:57.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:56 smithi097 ceph-mon[67407]: from='osd.6 ' entity='osd.6' 2024-04-12T22:06:57.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:56 smithi114 ceph-mon[58780]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-12T22:06:57.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:56 smithi114 ceph-mon[58780]: from='osd.6 [v2:172.21.15.114:6816/4257502981,v1:172.21.15.114:6817/4257502981]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-12T22:06:57.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:56 smithi114 ceph-mon[58780]: osdmap e145: 8 total, 7 up, 8 in 2024-04-12T22:06:57.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:56 smithi114 ceph-mon[58780]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-12T22:06:57.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:56 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:57.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:56 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:57.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:56 smithi114 ceph-mon[58780]: from='osd.6 ' entity='osd.6' 2024-04-12T22:06:58.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:57 smithi097 ceph-mon[64853]: pgmap v183: 161 pgs: 34 active+undersized, 19 active+undersized+degraded, 108 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 2 op/s; 76/630 objects degraded (12.063%) 2024-04-12T22:06:58.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:57 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:58.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:57 smithi097 ceph-mon[64853]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-12T22:06:58.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:57 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:58.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:57 smithi097 ceph-mon[64853]: osd.6 [v2:172.21.15.114:6816/4257502981,v1:172.21.15.114:6817/4257502981] boot 2024-04-12T22:06:58.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:57 smithi097 ceph-mon[64853]: osdmap e146: 8 total, 8 up, 8 in 2024-04-12T22:06:58.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:57 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-12T22:06:58.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:57 smithi097 ceph-mon[67407]: pgmap v183: 161 pgs: 34 active+undersized, 19 active+undersized+degraded, 108 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 2 op/s; 76/630 objects degraded (12.063%) 2024-04-12T22:06:58.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:57 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:58.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:57 smithi097 ceph-mon[67407]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-12T22:06:58.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:57 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:58.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:57 smithi097 ceph-mon[67407]: osd.6 [v2:172.21.15.114:6816/4257502981,v1:172.21.15.114:6817/4257502981] boot 2024-04-12T22:06:58.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:57 smithi097 ceph-mon[67407]: osdmap e146: 8 total, 8 up, 8 in 2024-04-12T22:06:58.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:57 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-12T22:06:58.204 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:57 smithi114 ceph-mon[58780]: pgmap v183: 161 pgs: 34 active+undersized, 19 active+undersized+degraded, 108 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 2 op/s; 76/630 objects degraded (12.063%) 2024-04-12T22:06:58.204 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:57 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:58.204 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:57 smithi114 ceph-mon[58780]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-12T22:06:58.205 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:57 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:06:58.205 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:57 smithi114 ceph-mon[58780]: osd.6 [v2:172.21.15.114:6816/4257502981,v1:172.21.15.114:6817/4257502981] boot 2024-04-12T22:06:58.205 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:57 smithi114 ceph-mon[58780]: osdmap e146: 8 total, 8 up, 8 in 2024-04-12T22:06:58.205 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:57 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-12T22:06:59.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:59 smithi097 ceph-mon[64853]: pgmap v185: 161 pgs: 10 peering, 27 active+undersized, 16 active+undersized+degraded, 108 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 39 op/s; 62/630 objects degraded (9.841%) 2024-04-12T22:06:59.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:59 smithi097 ceph-mon[64853]: osdmap e147: 8 total, 8 up, 8 in 2024-04-12T22:06:59.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:06:59 smithi097 ceph-mon[64853]: Health check update: Degraded data redundancy: 62/630 objects degraded (9.841%), 16 pgs degraded (PG_DEGRADED) 2024-04-12T22:06:59.660 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:06:59 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:06:59] "GET /metrics HTTP/1.1" 200 38472 "" "Prometheus/2.33.4" 2024-04-12T22:06:59.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:59 smithi097 ceph-mon[67407]: pgmap v185: 161 pgs: 10 peering, 27 active+undersized, 16 active+undersized+degraded, 108 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 39 op/s; 62/630 objects degraded (9.841%) 2024-04-12T22:06:59.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:59 smithi097 ceph-mon[67407]: osdmap e147: 8 total, 8 up, 8 in 2024-04-12T22:06:59.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:06:59 smithi097 ceph-mon[67407]: Health check update: Degraded data redundancy: 62/630 objects degraded (9.841%), 16 pgs degraded (PG_DEGRADED) 2024-04-12T22:06:59.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:59 smithi114 ceph-mon[58780]: pgmap v185: 161 pgs: 10 peering, 27 active+undersized, 16 active+undersized+degraded, 108 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 39 op/s; 62/630 objects degraded (9.841%) 2024-04-12T22:06:59.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:59 smithi114 ceph-mon[58780]: osdmap e147: 8 total, 8 up, 8 in 2024-04-12T22:06:59.663 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:06:59 smithi114 ceph-mon[58780]: Health check update: Degraded data redundancy: 62/630 objects degraded (9.841%), 16 pgs degraded (PG_DEGRADED) 2024-04-12T22:07:01.361 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:07:01 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:07:01.089+0000 7fe40fa5a700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (5 PGs are or would become offline) 2024-04-12T22:07:01.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:01 smithi097 ceph-mon[64853]: pgmap v187: 161 pgs: 10 peering, 27 active+undersized, 16 active+undersized+degraded, 108 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s; 62/630 objects degraded (9.841%) 2024-04-12T22:07:01.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:01 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:01.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:01 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:01.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:01 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:07:01.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:01 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:07:01.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:01 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:01.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:01 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:07:01.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:01 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:07:01.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:01 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:07:01.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:01 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:07:01.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:01 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-12T22:07:01.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:01 smithi097 ceph-mon[67407]: pgmap v187: 161 pgs: 10 peering, 27 active+undersized, 16 active+undersized+degraded, 108 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s; 62/630 objects degraded (9.841%) 2024-04-12T22:07:01.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:01 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:01.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:01 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:01.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:01 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:07:01.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:01 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:07:01.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:01 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:01.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:01 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:07:01.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:01 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:07:01.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:01 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:07:01.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:01 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:07:01.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:01 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-12T22:07:01.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:01 smithi114 ceph-mon[58780]: pgmap v187: 161 pgs: 10 peering, 27 active+undersized, 16 active+undersized+degraded, 108 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s; 62/630 objects degraded (9.841%) 2024-04-12T22:07:01.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:01 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:01.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:01 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:01.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:01 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:07:01.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:01 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:07:01.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:01 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:01.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:01 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:07:01.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:01 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:07:01.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:01 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:07:01.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:01 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:07:01.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:01 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-12T22:07:02.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:02 smithi097 ceph-mon[64853]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:07:02.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:02 smithi097 ceph-mon[64853]: pgmap v188: 161 pgs: 10 peering, 13 active+undersized, 7 active+undersized+degraded, 131 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 0 B/s wr, 182 op/s; 31/630 objects degraded (4.921%); 0 B/s, 4 objects/s recovering 2024-04-12T22:07:02.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:02 smithi097 ceph-mon[64853]: pgmap v189: 161 pgs: 10 peering, 13 active+undersized, 7 active+undersized+degraded, 131 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 113 KiB/s rd, 0 B/s wr, 188 op/s; 31/630 objects degraded (4.921%); 0 B/s, 4 objects/s recovering 2024-04-12T22:07:02.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:02 smithi097 ceph-mon[64853]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-12T22:07:02.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:02 smithi097 ceph-mon[64853]: Upgrade: unsafe to stop osd(s) at this time (5 PGs are or would become offline) 2024-04-12T22:07:02.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:02 smithi097 ceph-mon[67407]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:07:02.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:02 smithi097 ceph-mon[67407]: pgmap v188: 161 pgs: 10 peering, 13 active+undersized, 7 active+undersized+degraded, 131 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 0 B/s wr, 182 op/s; 31/630 objects degraded (4.921%); 0 B/s, 4 objects/s recovering 2024-04-12T22:07:02.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:02 smithi097 ceph-mon[67407]: pgmap v189: 161 pgs: 10 peering, 13 active+undersized, 7 active+undersized+degraded, 131 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 113 KiB/s rd, 0 B/s wr, 188 op/s; 31/630 objects degraded (4.921%); 0 B/s, 4 objects/s recovering 2024-04-12T22:07:02.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:02 smithi097 ceph-mon[67407]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-12T22:07:02.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:02 smithi097 ceph-mon[67407]: Upgrade: unsafe to stop osd(s) at this time (5 PGs are or would become offline) 2024-04-12T22:07:02.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:02 smithi114 ceph-mon[58780]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:07:02.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:02 smithi114 ceph-mon[58780]: pgmap v188: 161 pgs: 10 peering, 13 active+undersized, 7 active+undersized+degraded, 131 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 0 B/s wr, 182 op/s; 31/630 objects degraded (4.921%); 0 B/s, 4 objects/s recovering 2024-04-12T22:07:02.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:02 smithi114 ceph-mon[58780]: pgmap v189: 161 pgs: 10 peering, 13 active+undersized, 7 active+undersized+degraded, 131 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 113 KiB/s rd, 0 B/s wr, 188 op/s; 31/630 objects degraded (4.921%); 0 B/s, 4 objects/s recovering 2024-04-12T22:07:02.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:02 smithi114 ceph-mon[58780]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-12T22:07:02.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:02 smithi114 ceph-mon[58780]: Upgrade: unsafe to stop osd(s) at this time (5 PGs are or would become offline) 2024-04-12T22:07:03.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:03 smithi114 ceph-mon[58780]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 31/630 objects degraded (4.921%), 7 pgs degraded) 2024-04-12T22:07:03.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:03 smithi097 ceph-mon[64853]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 31/630 objects degraded (4.921%), 7 pgs degraded) 2024-04-12T22:07:03.910 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:03 smithi097 ceph-mon[67407]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 31/630 objects degraded (4.921%), 7 pgs degraded) 2024-04-12T22:07:04.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:04 smithi114 ceph-mon[58780]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 128 KiB/s rd, 0 B/s wr, 212 op/s; 0 B/s, 9 objects/s recovering 2024-04-12T22:07:04.910 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:04 smithi097 ceph-mon[67407]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 128 KiB/s rd, 0 B/s wr, 212 op/s; 0 B/s, 9 objects/s recovering 2024-04-12T22:07:04.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:04 smithi097 ceph-mon[64853]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 128 KiB/s rd, 0 B/s wr, 212 op/s; 0 B/s, 9 objects/s recovering 2024-04-12T22:07:06.742 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:06 smithi114 ceph-mon[58780]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 0 B/s wr, 164 op/s; 0 B/s, 8 objects/s recovering 2024-04-12T22:07:06.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:06 smithi097 ceph-mon[64853]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 0 B/s wr, 164 op/s; 0 B/s, 8 objects/s recovering 2024-04-12T22:07:06.910 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:06 smithi097 ceph-mon[67407]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 0 B/s wr, 164 op/s; 0 B/s, 8 objects/s recovering 2024-04-12T22:07:08.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:08 smithi114 ceph-mon[58780]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 0 B/s wr, 142 op/s; 0 B/s, 6 objects/s recovering 2024-04-12T22:07:08.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:08 smithi097 ceph-mon[64853]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 0 B/s wr, 142 op/s; 0 B/s, 6 objects/s recovering 2024-04-12T22:07:08.910 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:08 smithi097 ceph-mon[67407]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 0 B/s wr, 142 op/s; 0 B/s, 6 objects/s recovering 2024-04-12T22:07:09.660 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:07:09 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:07:09] "GET /metrics HTTP/1.1" 200 38472 "" "Prometheus/2.33.4" 2024-04-12T22:07:10.046 INFO:teuthology.orchestra.run.smithi097.stdout:true 2024-04-12T22:07:10.687 INFO:teuthology.orchestra.run.smithi097.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T22:07:10.687 INFO:teuthology.orchestra.run.smithi097.stdout:alertmanager.a smithi097 *:9093,9094 running (6m) 102s ago 11m 25.5M - 0.23.0 ba2b418f427c 7b4c491fe0a0 2024-04-12T22:07:10.687 INFO:teuthology.orchestra.run.smithi097.stdout:grafana.a smithi114 *:3000 error 14s ago 10m - - 2024-04-12T22:07:10.687 INFO:teuthology.orchestra.run.smithi097.stdout:iscsi.foo.smithi097.xndkxy smithi097 running (5m) 102s ago 9m 52.6M - 3.5 e1d6a67b021e 0436463f28cd 2024-04-12T22:07:10.688 INFO:teuthology.orchestra.run.smithi097.stdout:mgr.x smithi114 *:8443,9283,8765 running (4m) 14s ago 15m 451M - 19.0.0-2053-gb277feb2 668895944f0e b8f8df3bf928 2024-04-12T22:07:10.688 INFO:teuthology.orchestra.run.smithi097.stdout:mgr.y smithi097 *:8443,9283,8765 running (5m) 102s ago 17m 506M - 19.0.0-2053-gb277feb2 668895944f0e 488170dac11f 2024-04-12T22:07:10.688 INFO:teuthology.orchestra.run.smithi097.stdout:mon.a smithi097 running (4m) 102s ago 17m 50.6M 2048M 19.0.0-2053-gb277feb2 668895944f0e 3cfa16d3a85c 2024-04-12T22:07:10.688 INFO:teuthology.orchestra.run.smithi097.stdout:mon.b smithi114 running (3m) 14s ago 15m 40.5M 2048M 19.0.0-2053-gb277feb2 668895944f0e 71a2eff159a3 2024-04-12T22:07:10.688 INFO:teuthology.orchestra.run.smithi097.stdout:mon.c smithi097 running (4m) 102s ago 15m 39.3M 2048M 19.0.0-2053-gb277feb2 668895944f0e 69507ba88603 2024-04-12T22:07:10.688 INFO:teuthology.orchestra.run.smithi097.stdout:node-exporter.a smithi097 *:9100 running (6m) 102s ago 11m 17.7M - 1.3.1 1dbe0e931976 5edff9053b37 2024-04-12T22:07:10.688 INFO:teuthology.orchestra.run.smithi097.stdout:node-exporter.b smithi114 *:9100 running (5m) 14s ago 11m 16.9M - 1.3.1 1dbe0e931976 904afc45dc2f 2024-04-12T22:07:10.688 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0 smithi097 running (3m) 102s ago 15m 51.2M 2736M 19.0.0-2053-gb277feb2 668895944f0e 9d05fea4bf37 2024-04-12T22:07:10.688 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1 smithi097 running (2m) 102s ago 14m 43.7M 2736M 19.0.0-2053-gb277feb2 668895944f0e a9a527642889 2024-04-12T22:07:10.688 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2 smithi097 running (2m) 102s ago 14m 43.7M 2736M 19.0.0-2053-gb277feb2 668895944f0e 2e2297afff50 2024-04-12T22:07:10.688 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3 smithi097 running (107s) 102s ago 13m 12.6M 2736M 19.0.0-2053-gb277feb2 668895944f0e 415b4d3d85f2 2024-04-12T22:07:10.688 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4 smithi114 running (87s) 14s ago 13m 45.4M 2992M 19.0.0-2053-gb277feb2 668895944f0e fcd2e39a5866 2024-04-12T22:07:10.688 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5 smithi114 running (53s) 14s ago 13m 50.9M 2992M 19.0.0-2053-gb277feb2 668895944f0e 470c0780e428 2024-04-12T22:07:10.688 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6 smithi114 running (18s) 14s ago 12m 12.6M 2992M 19.0.0-2053-gb277feb2 668895944f0e 6b80aaeca08b 2024-04-12T22:07:10.688 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7 smithi114 running (12m) 14s ago 12m 60.0M 2992M 17.2.0 e1d6a67b021e f68340fecf2c 2024-04-12T22:07:10.689 INFO:teuthology.orchestra.run.smithi097.stdout:prometheus.a smithi114 *:9095 running (5m) 14s ago 11m 71.8M - 2.33.4 514e6a882f6e 0ad5437ed63f 2024-04-12T22:07:10.689 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.foo.smithi097.sggtkh smithi097 *:8000 running (10m) 102s ago 10m 73.0M - 17.2.0 e1d6a67b021e a39f9b2d8509 2024-04-12T22:07:10.689 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.foo.smithi114.ulfbyo smithi114 *:8000 running (10m) 14s ago 10m 73.7M - 17.2.0 e1d6a67b021e df79a48bdd59 2024-04-12T22:07:10.689 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.smpl.smithi097.exxkgl smithi097 *:80 running (10m) 102s ago 10m 72.9M - 17.2.0 e1d6a67b021e a036c58a0099 2024-04-12T22:07:10.689 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.smpl.smithi114.lrgvph smithi114 *:80 running (10m) 14s ago 10m 75.2M - 17.2.0 e1d6a67b021e 031a5fc24237 2024-04-12T22:07:10.946 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:10 smithi097 ceph-mon[64853]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 0 B/s wr, 159 op/s; 0 B/s, 6 objects/s recovering 2024-04-12T22:07:10.946 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:10 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:10.946 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:10 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:07:10.946 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:10 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:10.946 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:10 smithi097 ceph-mon[67407]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 0 B/s wr, 159 op/s; 0 B/s, 6 objects/s recovering 2024-04-12T22:07:10.947 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:10 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:10.947 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:10 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:07:10.947 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:10 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:11.078 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-12T22:07:11.078 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": { 2024-04-12T22:07:11.078 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 3 2024-04-12T22:07:11.078 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:07:11.078 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": { 2024-04-12T22:07:11.078 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 2 2024-04-12T22:07:11.078 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:07:11.078 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": { 2024-04-12T22:07:11.078 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-04-12T22:07:11.078 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 7 2024-04-12T22:07:11.078 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:07:11.079 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw": { 2024-04-12T22:07:11.079 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-12T22:07:11.079 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:07:11.079 INFO:teuthology.orchestra.run.smithi097.stdout: "overall": { 2024-04-12T22:07:11.079 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-04-12T22:07:11.079 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 12 2024-04-12T22:07:11.079 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-12T22:07:11.079 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-12T22:07:11.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:10 smithi114 ceph-mon[58780]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 0 B/s wr, 159 op/s; 0 B/s, 6 objects/s recovering 2024-04-12T22:07:11.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:10 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:11.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:10 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:07:11.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:10 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:11.405 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-12T22:07:11.405 INFO:teuthology.orchestra.run.smithi097.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a", 2024-04-12T22:07:11.405 INFO:teuthology.orchestra.run.smithi097.stdout: "in_progress": true, 2024-04-12T22:07:11.405 INFO:teuthology.orchestra.run.smithi097.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T22:07:11.406 INFO:teuthology.orchestra.run.smithi097.stdout: "services_complete": [ 2024-04-12T22:07:11.406 INFO:teuthology.orchestra.run.smithi097.stdout: "mon", 2024-04-12T22:07:11.406 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr" 2024-04-12T22:07:11.406 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-04-12T22:07:11.406 INFO:teuthology.orchestra.run.smithi097.stdout: "progress": "12/23 daemons upgraded", 2024-04-12T22:07:11.406 INFO:teuthology.orchestra.run.smithi097.stdout: "message": "Currently upgrading osd daemons", 2024-04-12T22:07:11.406 INFO:teuthology.orchestra.run.smithi097.stdout: "is_paused": false 2024-04-12T22:07:11.406 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-12T22:07:11.800 INFO:teuthology.orchestra.run.smithi097.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-12T22:07:11.801 INFO:teuthology.orchestra.run.smithi097.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-12T22:07:11.801 INFO:teuthology.orchestra.run.smithi097.stdout: daemon grafana.a on smithi114 is in error state 2024-04-12T22:07:12.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:11 smithi097 ceph-mon[64853]: from='client.34461 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:07:12.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:11 smithi097 ceph-mon[64853]: from='client.34467 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:07:12.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:11 smithi097 ceph-mon[64853]: from='client.34473 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:07:12.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:11 smithi097 ceph-mon[64853]: from='client.? 172.21.15.97:0/2419229200' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:07:12.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:11 smithi097 ceph-mon[67407]: from='client.34461 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:07:12.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:11 smithi097 ceph-mon[67407]: from='client.34467 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:07:12.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:11 smithi097 ceph-mon[67407]: from='client.34473 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:07:12.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:11 smithi097 ceph-mon[67407]: from='client.? 172.21.15.97:0/2419229200' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:07:12.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:11 smithi114 ceph-mon[58780]: from='client.34461 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:07:12.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:11 smithi114 ceph-mon[58780]: from='client.34467 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:07:12.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:11 smithi114 ceph-mon[58780]: from='client.34473 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:07:12.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:11 smithi114 ceph-mon[58780]: from='client.? 172.21.15.97:0/2419229200' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:07:13.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:12 smithi097 ceph-mon[64853]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 0 B/s wr, 141 op/s; 0 B/s, 3 objects/s recovering 2024-04-12T22:07:13.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:12 smithi097 ceph-mon[64853]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:07:13.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:12 smithi097 ceph-mon[64853]: from='client.44413 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:07:13.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:12 smithi097 ceph-mon[64853]: from='client.? 172.21.15.97:0/3049544301' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T22:07:13.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:12 smithi097 ceph-mon[67407]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 0 B/s wr, 141 op/s; 0 B/s, 3 objects/s recovering 2024-04-12T22:07:13.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:12 smithi097 ceph-mon[67407]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:07:13.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:12 smithi097 ceph-mon[67407]: from='client.44413 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:07:13.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:12 smithi097 ceph-mon[67407]: from='client.? 172.21.15.97:0/3049544301' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T22:07:13.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:12 smithi114 ceph-mon[58780]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 0 B/s wr, 141 op/s; 0 B/s, 3 objects/s recovering 2024-04-12T22:07:13.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:12 smithi114 ceph-mon[58780]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:07:13.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:12 smithi114 ceph-mon[58780]: from='client.44413 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:07:13.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:12 smithi114 ceph-mon[58780]: from='client.? 172.21.15.97:0/3049544301' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T22:07:15.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:14 smithi097 ceph-mon[64853]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 98 KiB/s rd, 0 B/s wr, 162 op/s; 0 B/s, 2 objects/s recovering 2024-04-12T22:07:15.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:14 smithi097 ceph-mon[67407]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 98 KiB/s rd, 0 B/s wr, 162 op/s; 0 B/s, 2 objects/s recovering 2024-04-12T22:07:15.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:14 smithi114 ceph-mon[58780]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 98 KiB/s rd, 0 B/s wr, 162 op/s; 0 B/s, 2 objects/s recovering 2024-04-12T22:07:16.969 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:16 smithi114 ceph-mon[58780]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-04-12T22:07:16.970 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:16 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-12T22:07:17.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:16 smithi097 ceph-mon[64853]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-04-12T22:07:17.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:16 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-12T22:07:17.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:16 smithi097 ceph-mon[67407]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-04-12T22:07:17.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:16 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-12T22:07:17.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:17 smithi114 ceph-mon[58780]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-12T22:07:17.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:17 smithi114 ceph-mon[58780]: Upgrade: osd.7 is safe to restart 2024-04-12T22:07:17.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:17 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:17.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:17 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-12T22:07:17.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:17 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:07:18.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:17 smithi097 ceph-mon[64853]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-12T22:07:18.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:17 smithi097 ceph-mon[64853]: Upgrade: osd.7 is safe to restart 2024-04-12T22:07:18.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:17 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:18.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:17 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-12T22:07:18.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:17 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:07:18.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:17 smithi097 ceph-mon[67407]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-12T22:07:18.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:17 smithi097 ceph-mon[67407]: Upgrade: osd.7 is safe to restart 2024-04-12T22:07:18.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:17 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:18.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:17 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-12T22:07:18.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:17 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:07:18.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:18 smithi114 ceph-mon[58780]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-04-12T22:07:18.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:18 smithi114 ceph-mon[58780]: Upgrade: Updating osd.7 2024-04-12T22:07:18.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:18 smithi114 ceph-mon[58780]: Deploying daemon osd.7 on smithi114 2024-04-12T22:07:19.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:18 smithi097 ceph-mon[64853]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-04-12T22:07:19.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:18 smithi097 ceph-mon[64853]: Upgrade: Updating osd.7 2024-04-12T22:07:19.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:18 smithi097 ceph-mon[64853]: Deploying daemon osd.7 on smithi114 2024-04-12T22:07:19.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:18 smithi097 ceph-mon[67407]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-04-12T22:07:19.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:18 smithi097 ceph-mon[67407]: Upgrade: Updating osd.7 2024-04-12T22:07:19.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:18 smithi097 ceph-mon[67407]: Deploying daemon osd.7 on smithi114 2024-04-12T22:07:19.660 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:07:19 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:07:19] "GET /metrics HTTP/1.1" 200 38477 "" "Prometheus/2.33.4" 2024-04-12T22:07:20.892 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:20 smithi114 ceph-mon[58780]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 110 op/s 2024-04-12T22:07:21.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:20 smithi097 ceph-mon[64853]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 110 op/s 2024-04-12T22:07:21.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:20 smithi097 ceph-mon[67407]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 110 op/s 2024-04-12T22:07:21.185 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:07:20 smithi114 systemd[1]: Stopping Ceph osd.7 for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:07:21.493 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:07:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[37937]: 2024-04-12T22:07:21.183+0000 7f1f8f3cf700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-12T22:07:21.493 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:07:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[37937]: 2024-04-12T22:07:21.183+0000 7f1f8f3cf700 -1 osd.7 147 *** Got signal Terminated *** 2024-04-12T22:07:21.493 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:07:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[37937]: 2024-04-12T22:07:21.183+0000 7f1f8f3cf700 -1 osd.7 147 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-12T22:07:22.058 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:21 smithi114 ceph-mon[58780]: osd.7 marked itself down and dead 2024-04-12T22:07:22.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:21 smithi097 ceph-mon[64853]: osd.7 marked itself down and dead 2024-04-12T22:07:22.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:21 smithi097 ceph-mon[67407]: osd.7 marked itself down and dead 2024-04-12T22:07:22.493 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:07:22 smithi114 podman[68848]: 2024-04-12 22:07:22.274032198 +0000 UTC m=+1.218900664 container died f68340fecf2c02d1f7ef729572ba57237b440133f8d7130630d6cde6c296d29f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, release=754, com.redhat.component=centos-stream-container, vcs-type=git, architecture=x86_64, GIT_CLEAN=True, version=8) 2024-04-12T22:07:22.814 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:07:22 smithi114 podman[68848]: 2024-04-12 22:07:22.709533509 +0000 UTC m=+1.654401961 container cleanup f68340fecf2c02d1f7ef729572ba57237b440133f8d7130630d6cde6c296d29f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7, com.redhat.component=centos-stream-container, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, RELEASE=HEAD, name=centos-stream, release=754, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.19.8, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-04-12T22:07:22.814 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:07:22 smithi114 bash[68848]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7 2024-04-12T22:07:23.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:22 smithi097 ceph-mon[64853]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 96 op/s 2024-04-12T22:07:23.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:22 smithi097 ceph-mon[64853]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:07:23.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:22 smithi097 ceph-mon[64853]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-12T22:07:23.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:22 smithi097 ceph-mon[64853]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-04-12T22:07:23.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:22 smithi097 ceph-mon[64853]: osdmap e148: 8 total, 7 up, 8 in 2024-04-12T22:07:23.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:22 smithi097 ceph-mon[67407]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 96 op/s 2024-04-12T22:07:23.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:22 smithi097 ceph-mon[67407]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:07:23.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:22 smithi097 ceph-mon[67407]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-12T22:07:23.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:22 smithi097 ceph-mon[67407]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-04-12T22:07:23.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:22 smithi097 ceph-mon[67407]: osdmap e148: 8 total, 7 up, 8 in 2024-04-12T22:07:23.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:22 smithi114 ceph-mon[58780]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 96 op/s 2024-04-12T22:07:23.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:22 smithi114 ceph-mon[58780]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:07:23.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:22 smithi114 ceph-mon[58780]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-12T22:07:23.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:22 smithi114 ceph-mon[58780]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-04-12T22:07:23.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:22 smithi114 ceph-mon[58780]: osdmap e148: 8 total, 7 up, 8 in 2024-04-12T22:07:23.243 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:07:22 smithi114 podman[68876]: 2024-04-12 22:07:22.886053606 +0000 UTC m=+0.609289067 container remove f68340fecf2c02d1f7ef729572ba57237b440133f8d7130630d6cde6c296d29f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, RELEASE=HEAD, distribution-scope=public, ceph=True, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-04-12T22:07:23.606 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:07:23 smithi114 podman[68969]: 2024-04-12 22:07:23.222128947 +0000 UTC m=+0.017547197 image pull 668895944f0ec64b67a5b7564961d2756f2966e5d4b9338516300dbec9984951 quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a 2024-04-12T22:07:23.606 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:07:23 smithi114 podman[68969]: 2024-04-12 22:07:23.32874865 +0000 UTC m=+0.124166884 container create fa787ba16c3ce22f3b3c087545ae89d0b06acbc7134a1c8d3d4faf1cb0ae5079 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7-deactivate, RELEASE=squid-b277feb, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-04-12T22:07:23.606 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:07:23 smithi114 podman[68969]: 2024-04-12 22:07:23.523061275 +0000 UTC m=+0.318479523 container init fa787ba16c3ce22f3b3c087545ae89d0b06acbc7134a1c8d3d4faf1cb0ae5079 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-b277feb, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-12T22:07:23.606 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:07:23 smithi114 podman[68969]: 2024-04-12 22:07:23.526396422 +0000 UTC m=+0.321814668 container start fa787ba16c3ce22f3b3c087545ae89d0b06acbc7134a1c8d3d4faf1cb0ae5079 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7-deactivate, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, RELEASE=squid-b277feb) 2024-04-12T22:07:23.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:23 smithi114 ceph-mon[58780]: osdmap e149: 8 total, 7 up, 8 in 2024-04-12T22:07:23.993 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:07:23 smithi114 podman[68969]: 2024-04-12 22:07:23.606066644 +0000 UTC m=+0.401484883 container attach fa787ba16c3ce22f3b3c087545ae89d0b06acbc7134a1c8d3d4faf1cb0ae5079 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-b277feb, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-12T22:07:23.993 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:07:23 smithi114 podman[68969]: 2024-04-12 22:07:23.825332357 +0000 UTC m=+0.620750591 container died fa787ba16c3ce22f3b3c087545ae89d0b06acbc7134a1c8d3d4faf1cb0ae5079 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-b277feb, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409) 2024-04-12T22:07:24.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:23 smithi097 ceph-mon[64853]: osdmap e149: 8 total, 7 up, 8 in 2024-04-12T22:07:24.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:23 smithi097 ceph-mon[67407]: osdmap e149: 8 total, 7 up, 8 in 2024-04-12T22:07:24.832 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:07:24 smithi114 podman[68995]: 2024-04-12 22:07:24.526716842 +0000 UTC m=+0.691908793 container remove fa787ba16c3ce22f3b3c087545ae89d0b06acbc7134a1c8d3d4faf1cb0ae5079 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7-deactivate, RELEASE=squid-b277feb, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409) 2024-04-12T22:07:24.832 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:07:24 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@osd.7.service: Deactivated successfully. 2024-04-12T22:07:24.832 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:07:24 smithi114 systemd[1]: Stopped Ceph osd.7 for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:07:24.832 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:07:24 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@osd.7.service: Consumed 10.545s CPU time. 2024-04-12T22:07:24.832 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:07:24 smithi114 systemd[1]: Starting Ceph osd.7 for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:07:25.155 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:24 smithi114 ceph-mon[58780]: pgmap v202: 161 pgs: 35 peering, 11 stale+active+clean, 115 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:07:25.155 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:24 smithi114 ceph-mon[58780]: Health check failed: Reduced data availability: 4 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-04-12T22:07:25.155 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:24 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:25.155 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:24 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:07:25.156 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:07:24 smithi114 podman[69099]: 2024-04-12 22:07:24.953100749 +0000 UTC m=+0.099281192 container create 5e4af2db5c972425e69f3898f9c56c1ac544b63dd94d951969bda4a0fca720d6 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7-activate, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-b277feb, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-12T22:07:25.156 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:07:24 smithi114 podman[69099]: 2024-04-12 22:07:24.871426611 +0000 UTC m=+0.017607067 image pull 668895944f0ec64b67a5b7564961d2756f2966e5d4b9338516300dbec9984951 quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a 2024-04-12T22:07:25.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:24 smithi097 ceph-mon[64853]: pgmap v202: 161 pgs: 35 peering, 11 stale+active+clean, 115 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:07:25.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:24 smithi097 ceph-mon[64853]: Health check failed: Reduced data availability: 4 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-04-12T22:07:25.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:24 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:25.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:24 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:07:25.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:24 smithi097 ceph-mon[67407]: pgmap v202: 161 pgs: 35 peering, 11 stale+active+clean, 115 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:07:25.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:24 smithi097 ceph-mon[67407]: Health check failed: Reduced data availability: 4 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-04-12T22:07:25.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:24 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:25.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:24 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:07:25.493 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:07:25 smithi114 podman[69099]: 2024-04-12 22:07:25.155519445 +0000 UTC m=+0.301699888 container init 5e4af2db5c972425e69f3898f9c56c1ac544b63dd94d951969bda4a0fca720d6 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7-activate, GIT_CLEAN=True, RELEASE=squid-b277feb, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0) 2024-04-12T22:07:25.493 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:07:25 smithi114 podman[69099]: 2024-04-12 22:07:25.158647003 +0000 UTC m=+0.304827446 container start 5e4af2db5c972425e69f3898f9c56c1ac544b63dd94d951969bda4a0fca720d6 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7-activate, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240409, RELEASE=squid-b277feb, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-12T22:07:25.493 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:07:25 smithi114 podman[69099]: 2024-04-12 22:07:25.230580936 +0000 UTC m=+0.376761408 container attach 5e4af2db5c972425e69f3898f9c56c1ac544b63dd94d951969bda4a0fca720d6 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7-activate, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-b277feb, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-12T22:07:25.926 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:07:25 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7-activate[69114]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-12T22:07:25.927 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:07:25 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7-activate[69114]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-04-12T22:07:25.927 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:07:25 smithi114 bash[69099]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-12T22:07:25.927 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:07:25 smithi114 bash[69099]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-04-12T22:07:25.927 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:07:25 smithi114 bash[69099]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-04-12T22:07:25.927 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:07:25 smithi114 bash[69099]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-04-12T22:07:25.927 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:07:25 smithi114 bash[69099]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-04-12T22:07:25.927 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:07:25 smithi114 bash[69099]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-12T22:07:25.927 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:07:25 smithi114 bash[69099]: --> ceph-volume raw activate successful for osd ID: 7 2024-04-12T22:07:25.927 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:07:25 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7-activate[69114]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-04-12T22:07:25.927 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:07:25 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7-activate[69114]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-04-12T22:07:25.927 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:07:25 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7-activate[69114]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-04-12T22:07:25.927 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:07:25 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7-activate[69114]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-12T22:07:25.927 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:07:25 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7-activate[69114]: --> ceph-volume raw activate successful for osd ID: 7 2024-04-12T22:07:25.928 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:07:25 smithi114 podman[69099]: 2024-04-12 22:07:25.616741159 +0000 UTC m=+0.762921596 container died 5e4af2db5c972425e69f3898f9c56c1ac544b63dd94d951969bda4a0fca720d6 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7-activate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=squid-b277feb, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-12T22:07:26.718 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:07:26 smithi114 podman[69245]: 2024-04-12 22:07:26.326408809 +0000 UTC m=+0.703734824 container remove 5e4af2db5c972425e69f3898f9c56c1ac544b63dd94d951969bda4a0fca720d6 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7-activate, org.label-schema.license=GPLv2, RELEASE=squid-b277feb, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-12T22:07:26.718 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:07:26 smithi114 podman[69308]: 2024-04-12 22:07:26.444357266 +0000 UTC m=+0.017736017 image pull 668895944f0ec64b67a5b7564961d2756f2966e5d4b9338516300dbec9984951 quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a 2024-04-12T22:07:26.719 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:07:26 smithi114 podman[69308]: 2024-04-12 22:07:26.552186431 +0000 UTC m=+0.125565167 container create 1c3e102eaf74e3bde0f201adfbb7f20b6c4eb14e8de83c30cd00bd066d17da36 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-b277feb, maintainer=Guillaume Abrioux ) 2024-04-12T22:07:26.979 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:26 smithi114 ceph-mon[58780]: pgmap v203: 161 pgs: 35 peering, 11 stale+active+clean, 115 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:07:26.979 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:07:26 smithi114 podman[69308]: 2024-04-12 22:07:26.888138982 +0000 UTC m=+0.461517715 container init 1c3e102eaf74e3bde0f201adfbb7f20b6c4eb14e8de83c30cd00bd066d17da36 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, RELEASE=squid-b277feb) 2024-04-12T22:07:26.980 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:07:26 smithi114 podman[69308]: 2024-04-12 22:07:26.891180548 +0000 UTC m=+0.464559282 container start 1c3e102eaf74e3bde0f201adfbb7f20b6c4eb14e8de83c30cd00bd066d17da36 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7, org.label-schema.schema-version=1.0, RELEASE=squid-b277feb, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-12T22:07:26.980 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:07:26 smithi114 ceph-osd[69327]: -- 172.21.15.114:0/3724437647 <== mon.2 v2:172.21.15.114:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f4f490b680 con 0x55f4f48d5400 2024-04-12T22:07:27.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:26 smithi097 ceph-mon[64853]: pgmap v203: 161 pgs: 35 peering, 11 stale+active+clean, 115 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:07:27.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:26 smithi097 ceph-mon[67407]: pgmap v203: 161 pgs: 35 peering, 11 stale+active+clean, 115 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:07:27.243 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:07:26 smithi114 bash[69308]: 1c3e102eaf74e3bde0f201adfbb7f20b6c4eb14e8de83c30cd00bd066d17da36 2024-04-12T22:07:27.243 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:07:26 smithi114 systemd[1]: Started Ceph osd.7 for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:07:27.742 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:07:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:07:27.438+0000 7fc43a7eb7c0 -1 Falling back to public interface 2024-04-12T22:07:28.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:28 smithi114 ceph-mon[58780]: pgmap v204: 161 pgs: 27 active+undersized, 19 active+undersized+degraded, 23 peering, 92 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 48/630 objects degraded (7.619%) 2024-04-12T22:07:28.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:28 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:28.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:28 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:28.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:28 smithi114 ceph-mon[58780]: Health check failed: Degraded data redundancy: 48/630 objects degraded (7.619%), 19 pgs degraded (PG_DEGRADED) 2024-04-12T22:07:28.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:28 smithi097 ceph-mon[64853]: pgmap v204: 161 pgs: 27 active+undersized, 19 active+undersized+degraded, 23 peering, 92 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 48/630 objects degraded (7.619%) 2024-04-12T22:07:28.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:28 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:28.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:28 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:28.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:28 smithi097 ceph-mon[64853]: Health check failed: Degraded data redundancy: 48/630 objects degraded (7.619%), 19 pgs degraded (PG_DEGRADED) 2024-04-12T22:07:28.410 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:28 smithi097 ceph-mon[67407]: pgmap v204: 161 pgs: 27 active+undersized, 19 active+undersized+degraded, 23 peering, 92 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 48/630 objects degraded (7.619%) 2024-04-12T22:07:28.410 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:28 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:28.410 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:28 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:28.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:28 smithi097 ceph-mon[67407]: Health check failed: Degraded data redundancy: 48/630 objects degraded (7.619%), 19 pgs degraded (PG_DEGRADED) 2024-04-12T22:07:29.144 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:29 smithi114 ceph-mon[58780]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 5 pgs peering) 2024-04-12T22:07:29.410 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:29 smithi097 ceph-mon[67407]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 5 pgs peering) 2024-04-12T22:07:29.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:29 smithi097 ceph-mon[64853]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 5 pgs peering) 2024-04-12T22:07:29.410 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:07:29 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:07:29] "GET /metrics HTTP/1.1" 200 38620 "" "Prometheus/2.33.4" 2024-04-12T22:07:30.153 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:30 smithi114 ceph-mon[58780]: pgmap v205: 161 pgs: 39 active+undersized, 30 active+undersized+degraded, 92 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 85/630 objects degraded (13.492%) 2024-04-12T22:07:30.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:30 smithi097 ceph-mon[64853]: pgmap v205: 161 pgs: 39 active+undersized, 30 active+undersized+degraded, 92 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 85/630 objects degraded (13.492%) 2024-04-12T22:07:30.410 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:30 smithi097 ceph-mon[67407]: pgmap v205: 161 pgs: 39 active+undersized, 30 active+undersized+degraded, 92 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 85/630 objects degraded (13.492%) 2024-04-12T22:07:30.495 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:07:30 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:07:30.152+0000 7fc43a7eb7c0 -1 osd.7 0 read_superblock omap replica is missing. 2024-04-12T22:07:30.495 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:07:30 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:07:30.189+0000 7fc43a7eb7c0 -1 osd.7 147 log_to_monitors true 2024-04-12T22:07:31.410 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:31 smithi097 ceph-mon[67407]: from='osd.7 [v2:172.21.15.114:6824/2506245008,v1:172.21.15.114:6825/2506245008]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-12T22:07:31.410 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:31 smithi097 ceph-mon[67407]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-12T22:07:31.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:31 smithi097 ceph-mon[64853]: from='osd.7 [v2:172.21.15.114:6824/2506245008,v1:172.21.15.114:6825/2506245008]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-12T22:07:31.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:31 smithi097 ceph-mon[64853]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-12T22:07:31.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:31 smithi114 ceph-mon[58780]: from='osd.7 [v2:172.21.15.114:6824/2506245008,v1:172.21.15.114:6825/2506245008]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-12T22:07:31.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:31 smithi114 ceph-mon[58780]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-12T22:07:31.993 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:07:31 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:07:31.565+0000 7fc427c3b700 -1 osd.7 147 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T22:07:32.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:32 smithi097 ceph-mon[64853]: pgmap v206: 161 pgs: 39 active+undersized, 30 active+undersized+degraded, 92 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 85/630 objects degraded (13.492%) 2024-04-12T22:07:32.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:32 smithi097 ceph-mon[64853]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:07:32.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:32 smithi097 ceph-mon[64853]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-12T22:07:32.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:32 smithi097 ceph-mon[64853]: osdmap e150: 8 total, 7 up, 8 in 2024-04-12T22:07:32.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:32 smithi097 ceph-mon[64853]: from='osd.7 [v2:172.21.15.114:6824/2506245008,v1:172.21.15.114:6825/2506245008]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-12T22:07:32.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:32 smithi097 ceph-mon[64853]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-12T22:07:32.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:32 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:32.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:32 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:32.411 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:32 smithi097 ceph-mon[64853]: from='osd.7 ' entity='osd.7' 2024-04-12T22:07:32.411 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:32 smithi097 ceph-mon[67407]: pgmap v206: 161 pgs: 39 active+undersized, 30 active+undersized+degraded, 92 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 85/630 objects degraded (13.492%) 2024-04-12T22:07:32.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:32 smithi097 ceph-mon[67407]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:07:32.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:32 smithi097 ceph-mon[67407]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-12T22:07:32.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:32 smithi097 ceph-mon[67407]: osdmap e150: 8 total, 7 up, 8 in 2024-04-12T22:07:32.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:32 smithi097 ceph-mon[67407]: from='osd.7 [v2:172.21.15.114:6824/2506245008,v1:172.21.15.114:6825/2506245008]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-12T22:07:32.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:32 smithi097 ceph-mon[67407]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-12T22:07:32.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:32 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:32.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:32 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:32.412 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:32 smithi097 ceph-mon[67407]: from='osd.7 ' entity='osd.7' 2024-04-12T22:07:32.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:32 smithi114 ceph-mon[58780]: pgmap v206: 161 pgs: 39 active+undersized, 30 active+undersized+degraded, 92 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 85/630 objects degraded (13.492%) 2024-04-12T22:07:32.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:32 smithi114 ceph-mon[58780]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:07:32.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:32 smithi114 ceph-mon[58780]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-12T22:07:32.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:32 smithi114 ceph-mon[58780]: osdmap e150: 8 total, 7 up, 8 in 2024-04-12T22:07:32.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:32 smithi114 ceph-mon[58780]: from='osd.7 [v2:172.21.15.114:6824/2506245008,v1:172.21.15.114:6825/2506245008]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-12T22:07:32.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:32 smithi114 ceph-mon[58780]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-12T22:07:32.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:32 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:32.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:32 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:32.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:32 smithi114 ceph-mon[58780]: from='osd.7 ' entity='osd.7' 2024-04-12T22:07:33.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:33 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:33.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:33 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:33.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:33 smithi114 ceph-mon[58780]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-12T22:07:33.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:33 smithi114 ceph-mon[58780]: osd.7 [v2:172.21.15.114:6824/2506245008,v1:172.21.15.114:6825/2506245008] boot 2024-04-12T22:07:33.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:33 smithi114 ceph-mon[58780]: osdmap e151: 8 total, 8 up, 8 in 2024-04-12T22:07:33.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:33 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-12T22:07:33.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:33 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:33.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:33 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:33.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:33 smithi097 ceph-mon[64853]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-12T22:07:33.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:33 smithi097 ceph-mon[64853]: osd.7 [v2:172.21.15.114:6824/2506245008,v1:172.21.15.114:6825/2506245008] boot 2024-04-12T22:07:33.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:33 smithi097 ceph-mon[64853]: osdmap e151: 8 total, 8 up, 8 in 2024-04-12T22:07:33.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:33 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-12T22:07:33.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:33 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:33.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:33 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:33.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:33 smithi097 ceph-mon[67407]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-12T22:07:33.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:33 smithi097 ceph-mon[67407]: osd.7 [v2:172.21.15.114:6824/2506245008,v1:172.21.15.114:6825/2506245008] boot 2024-04-12T22:07:33.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:33 smithi097 ceph-mon[67407]: osdmap e151: 8 total, 8 up, 8 in 2024-04-12T22:07:33.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:33 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-12T22:07:34.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:34 smithi114 ceph-mon[58780]: pgmap v209: 161 pgs: 39 active+undersized, 30 active+undersized+degraded, 92 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 85/630 objects degraded (13.492%) 2024-04-12T22:07:34.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:34 smithi114 ceph-mon[58780]: osdmap e152: 8 total, 8 up, 8 in 2024-04-12T22:07:34.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:34 smithi097 ceph-mon[64853]: pgmap v209: 161 pgs: 39 active+undersized, 30 active+undersized+degraded, 92 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 85/630 objects degraded (13.492%) 2024-04-12T22:07:34.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:34 smithi097 ceph-mon[64853]: osdmap e152: 8 total, 8 up, 8 in 2024-04-12T22:07:34.910 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:34 smithi097 ceph-mon[67407]: pgmap v209: 161 pgs: 39 active+undersized, 30 active+undersized+degraded, 92 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 85/630 objects degraded (13.492%) 2024-04-12T22:07:34.910 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:34 smithi097 ceph-mon[67407]: osdmap e152: 8 total, 8 up, 8 in 2024-04-12T22:07:35.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:35 smithi097 ceph-mon[64853]: Health check update: Degraded data redundancy: 85/630 objects degraded (13.492%), 30 pgs degraded (PG_DEGRADED) 2024-04-12T22:07:35.910 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:35 smithi097 ceph-mon[67407]: Health check update: Degraded data redundancy: 85/630 objects degraded (13.492%), 30 pgs degraded (PG_DEGRADED) 2024-04-12T22:07:35.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:35 smithi114 ceph-mon[58780]: Health check update: Degraded data redundancy: 85/630 objects degraded (13.492%), 30 pgs degraded (PG_DEGRADED) 2024-04-12T22:07:36.910 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[67407]: pgmap v211: 161 pgs: 39 active+undersized, 30 active+undersized+degraded, 92 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 85/630 objects degraded (13.492%) 2024-04-12T22:07:36.910 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:36.910 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:36.910 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:07:36.910 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:07:36.910 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[67407]: pgmap v212: 161 pgs: 39 active+undersized, 30 active+undersized+degraded, 92 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 85/630 objects degraded (13.492%) 2024-04-12T22:07:36.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[67407]: pgmap v213: 161 pgs: 39 active+undersized, 30 active+undersized+degraded, 92 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 85/630 objects degraded (13.492%) 2024-04-12T22:07:36.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:36.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:07:36.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:07:36.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:07:36.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:07:36.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:07:36.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[67407]: Upgrade: Setting container_image for all osd 2024-04-12T22:07:36.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:36.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-04-12T22:07:36.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-04-12T22:07:36.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-04-12T22:07:36.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-04-12T22:07:36.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-04-12T22:07:36.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-04-12T22:07:36.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-04-12T22:07:36.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-04-12T22:07:36.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-04-12T22:07:36.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[64853]: pgmap v211: 161 pgs: 39 active+undersized, 30 active+undersized+degraded, 92 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 85/630 objects degraded (13.492%) 2024-04-12T22:07:36.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:36.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:36.912 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:07:36.913 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:07:36.913 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[64853]: pgmap v212: 161 pgs: 39 active+undersized, 30 active+undersized+degraded, 92 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 85/630 objects degraded (13.492%) 2024-04-12T22:07:36.913 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[64853]: pgmap v213: 161 pgs: 39 active+undersized, 30 active+undersized+degraded, 92 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 85/630 objects degraded (13.492%) 2024-04-12T22:07:36.913 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:36.913 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:07:36.913 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:07:36.913 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:07:36.913 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:07:36.913 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:07:36.913 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[64853]: Upgrade: Setting container_image for all osd 2024-04-12T22:07:36.913 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:36.913 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-04-12T22:07:36.913 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-04-12T22:07:36.913 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-04-12T22:07:36.913 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-04-12T22:07:36.914 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-04-12T22:07:36.914 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-04-12T22:07:36.914 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-04-12T22:07:36.914 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-04-12T22:07:36.914 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-04-12T22:07:36.914 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-04-12T22:07:36.914 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-04-12T22:07:36.914 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-04-12T22:07:36.914 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-04-12T22:07:36.914 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-04-12T22:07:36.915 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-04-12T22:07:36.915 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-04-12T22:07:36.915 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[67407]: Upgrade: Setting require_osd_release to 19 squid 2024-04-12T22:07:36.915 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-04-12T22:07:36.915 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-04-12T22:07:36.915 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-04-12T22:07:36.915 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-04-12T22:07:36.915 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-04-12T22:07:36.915 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-04-12T22:07:36.915 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-04-12T22:07:36.915 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-04-12T22:07:36.915 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[64853]: Upgrade: Setting require_osd_release to 19 squid 2024-04-12T22:07:36.916 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:36 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-04-12T22:07:36.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:36 smithi114 ceph-mon[58780]: pgmap v211: 161 pgs: 39 active+undersized, 30 active+undersized+degraded, 92 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 85/630 objects degraded (13.492%) 2024-04-12T22:07:36.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:36 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:36.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:36 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:36.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:36 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:07:36.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:36 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:07:36.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:36 smithi114 ceph-mon[58780]: pgmap v212: 161 pgs: 39 active+undersized, 30 active+undersized+degraded, 92 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 85/630 objects degraded (13.492%) 2024-04-12T22:07:36.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:36 smithi114 ceph-mon[58780]: pgmap v213: 161 pgs: 39 active+undersized, 30 active+undersized+degraded, 92 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 85/630 objects degraded (13.492%) 2024-04-12T22:07:36.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:36 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:36.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:36 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:07:36.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:36 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:07:36.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:36 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:07:36.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:36 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:07:36.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:36 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:07:36.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:36 smithi114 ceph-mon[58780]: Upgrade: Setting container_image for all osd 2024-04-12T22:07:36.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:36 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:36.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:36 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-04-12T22:07:36.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:36 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-04-12T22:07:36.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:36 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-04-12T22:07:36.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:36 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-04-12T22:07:36.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:36 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-04-12T22:07:36.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:36 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-04-12T22:07:36.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:36 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-04-12T22:07:36.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:36 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-04-12T22:07:36.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:36 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-04-12T22:07:36.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:36 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-04-12T22:07:36.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:36 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-04-12T22:07:36.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:36 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-04-12T22:07:36.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:36 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-04-12T22:07:36.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:36 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-04-12T22:07:36.996 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:36 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-04-12T22:07:36.996 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:36 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-04-12T22:07:36.996 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:36 smithi114 ceph-mon[58780]: Upgrade: Setting require_osd_release to 19 squid 2024-04-12T22:07:36.996 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:36 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-04-12T22:07:37.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:37 smithi097 ceph-mon[64853]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-04-12T22:07:37.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:37 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-04-12T22:07:37.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:37 smithi097 ceph-mon[64853]: osdmap e153: 8 total, 8 up, 8 in 2024-04-12T22:07:37.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:37 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:07:37.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:37 smithi097 ceph-mon[64853]: Upgrade: Setting container_image for all mds 2024-04-12T22:07:37.750 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:37 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:37.751 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:37 smithi097 ceph-mon[67407]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-04-12T22:07:37.751 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:37 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-04-12T22:07:37.751 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:37 smithi097 ceph-mon[67407]: osdmap e153: 8 total, 8 up, 8 in 2024-04-12T22:07:37.751 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:37 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:07:37.751 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:37 smithi097 ceph-mon[67407]: Upgrade: Setting container_image for all mds 2024-04-12T22:07:37.751 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:37 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:37.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:37 smithi114 ceph-mon[58780]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-04-12T22:07:37.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:37 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-04-12T22:07:37.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:37 smithi114 ceph-mon[58780]: osdmap e153: 8 total, 8 up, 8 in 2024-04-12T22:07:37.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:37 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:07:37.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:37 smithi114 ceph-mon[58780]: Upgrade: Setting container_image for all mds 2024-04-12T22:07:37.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:37 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:38.910 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:38 smithi097 ceph-mon[67407]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-12T22:07:38.910 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:38 smithi097 ceph-mon[67407]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 85/630 objects degraded (13.492%), 30 pgs degraded) 2024-04-12T22:07:38.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:38 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:38.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:38 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi097.sggtkh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-12T22:07:38.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:38 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:07:38.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:38 smithi097 ceph-mon[64853]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-12T22:07:38.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:38 smithi097 ceph-mon[64853]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 85/630 objects degraded (13.492%), 30 pgs degraded) 2024-04-12T22:07:38.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:38 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:38.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:38 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi097.sggtkh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-12T22:07:38.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:38 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:07:38.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:38 smithi114 ceph-mon[58780]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-12T22:07:38.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:38 smithi114 ceph-mon[58780]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 85/630 objects degraded (13.492%), 30 pgs degraded) 2024-04-12T22:07:38.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:38 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:38.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:38 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi097.sggtkh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-12T22:07:38.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:38 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:07:39.375 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:07:39 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:07:39] "GET /metrics HTTP/1.1" 200 38620 "" "Prometheus/2.33.4" 2024-04-12T22:07:39.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:39 smithi097 ceph-mon[64853]: Upgrade: Updating rgw.foo.smithi097.sggtkh (1/4) 2024-04-12T22:07:39.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:39 smithi097 ceph-mon[64853]: Deploying daemon rgw.foo.smithi097.sggtkh on smithi097 2024-04-12T22:07:39.910 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:39 smithi097 ceph-mon[67407]: Upgrade: Updating rgw.foo.smithi097.sggtkh (1/4) 2024-04-12T22:07:39.910 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:39 smithi097 ceph-mon[67407]: Deploying daemon rgw.foo.smithi097.sggtkh on smithi097 2024-04-12T22:07:39.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:39 smithi114 ceph-mon[58780]: Upgrade: Updating rgw.foo.smithi097.sggtkh (1/4) 2024-04-12T22:07:39.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:39 smithi114 ceph-mon[58780]: Deploying daemon rgw.foo.smithi097.sggtkh on smithi097 2024-04-12T22:07:40.869 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:40 smithi097 ceph-mon[64853]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 842 B/s rd, 0 op/s 2024-04-12T22:07:40.869 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:40 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:40.870 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:40 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:07:40.870 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:40 smithi097 ceph-mon[67407]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 842 B/s rd, 0 op/s 2024-04-12T22:07:40.870 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:40 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:40.870 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:40 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:07:41.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:40 smithi114 ceph-mon[58780]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 842 B/s rd, 0 op/s 2024-04-12T22:07:41.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:40 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:41.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:40 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:07:42.170 INFO:teuthology.orchestra.run.smithi097.stdout:true 2024-04-12T22:07:42.940 INFO:teuthology.orchestra.run.smithi097.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T22:07:42.940 INFO:teuthology.orchestra.run.smithi097.stdout:alertmanager.a smithi097 *:9093,9094 running (6m) 2m ago 11m 25.5M - 0.23.0 ba2b418f427c 7b4c491fe0a0 2024-04-12T22:07:42.940 INFO:teuthology.orchestra.run.smithi097.stdout:grafana.a smithi114 *:3000 error 11s ago 11m - - 2024-04-12T22:07:42.940 INFO:teuthology.orchestra.run.smithi097.stdout:iscsi.foo.smithi097.xndkxy smithi097 running (5m) 2m ago 10m 52.6M - 3.5 e1d6a67b021e 0436463f28cd 2024-04-12T22:07:42.940 INFO:teuthology.orchestra.run.smithi097.stdout:mgr.x smithi114 *:8443,9283,8765 running (5m) 11s ago 16m 452M - 19.0.0-2053-gb277feb2 668895944f0e b8f8df3bf928 2024-04-12T22:07:42.940 INFO:teuthology.orchestra.run.smithi097.stdout:mgr.y smithi097 *:8443,9283,8765 running (6m) 2m ago 17m 506M - 19.0.0-2053-gb277feb2 668895944f0e 488170dac11f 2024-04-12T22:07:42.940 INFO:teuthology.orchestra.run.smithi097.stdout:mon.a smithi097 running (5m) 2m ago 17m 50.6M 2048M 19.0.0-2053-gb277feb2 668895944f0e 3cfa16d3a85c 2024-04-12T22:07:42.940 INFO:teuthology.orchestra.run.smithi097.stdout:mon.b smithi114 running (4m) 11s ago 16m 40.5M 2048M 19.0.0-2053-gb277feb2 668895944f0e 71a2eff159a3 2024-04-12T22:07:42.940 INFO:teuthology.orchestra.run.smithi097.stdout:mon.c smithi097 running (4m) 2m ago 16m 39.3M 2048M 19.0.0-2053-gb277feb2 668895944f0e 69507ba88603 2024-04-12T22:07:42.941 INFO:teuthology.orchestra.run.smithi097.stdout:node-exporter.a smithi097 *:9100 running (6m) 2m ago 12m 17.7M - 1.3.1 1dbe0e931976 5edff9053b37 2024-04-12T22:07:42.941 INFO:teuthology.orchestra.run.smithi097.stdout:node-exporter.b smithi114 *:9100 running (6m) 11s ago 12m 17.1M - 1.3.1 1dbe0e931976 904afc45dc2f 2024-04-12T22:07:42.941 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0 smithi097 running (3m) 2m ago 15m 51.2M 2736M 19.0.0-2053-gb277feb2 668895944f0e 9d05fea4bf37 2024-04-12T22:07:42.941 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1 smithi097 running (3m) 2m ago 15m 43.7M 2736M 19.0.0-2053-gb277feb2 668895944f0e a9a527642889 2024-04-12T22:07:42.941 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2 smithi097 running (2m) 2m ago 14m 43.7M 2736M 19.0.0-2053-gb277feb2 668895944f0e 2e2297afff50 2024-04-12T22:07:42.941 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3 smithi097 running (2m) 2m ago 14m 12.6M 2736M 19.0.0-2053-gb277feb2 668895944f0e 415b4d3d85f2 2024-04-12T22:07:42.941 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4 smithi114 running (2m) 11s ago 14m 49.7M 2992M 19.0.0-2053-gb277feb2 668895944f0e fcd2e39a5866 2024-04-12T22:07:42.941 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5 smithi114 running (85s) 11s ago 13m 54.0M 2992M 19.0.0-2053-gb277feb2 668895944f0e 470c0780e428 2024-04-12T22:07:42.941 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6 smithi114 running (51s) 11s ago 13m 54.7M 2992M 19.0.0-2053-gb277feb2 668895944f0e 6b80aaeca08b 2024-04-12T22:07:42.941 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7 smithi114 running (16s) 11s ago 13m 11.7M 2992M 19.0.0-2053-gb277feb2 668895944f0e 1c3e102eaf74 2024-04-12T22:07:42.941 INFO:teuthology.orchestra.run.smithi097.stdout:prometheus.a smithi114 *:9095 running (5m) 11s ago 12m 71.9M - 2.33.4 514e6a882f6e 0ad5437ed63f 2024-04-12T22:07:42.941 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.foo.smithi097.sggtkh smithi097 *:8000 starting - - - - 2024-04-12T22:07:42.941 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.foo.smithi114.ulfbyo smithi114 *:8000 running (10m) 11s ago 10m 73.9M - 17.2.0 e1d6a67b021e df79a48bdd59 2024-04-12T22:07:42.941 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.smpl.smithi097.exxkgl smithi097 *:80 running (10m) 2m ago 10m 72.9M - 17.2.0 e1d6a67b021e a036c58a0099 2024-04-12T22:07:42.941 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.smpl.smithi114.lrgvph smithi114 *:80 running (10m) 11s ago 10m 75.4M - 17.2.0 e1d6a67b021e 031a5fc24237 2024-04-12T22:07:43.111 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:42 smithi097 ceph-mon[67407]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:07:43.111 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:42 smithi097 ceph-mon[67407]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 764 B/s rd, 0 op/s 2024-04-12T22:07:43.111 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:42 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:43.111 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:42 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:43.111 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:42 smithi097 ceph-mon[64853]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:07:43.111 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:42 smithi097 ceph-mon[64853]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 764 B/s rd, 0 op/s 2024-04-12T22:07:43.111 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:42 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:43.112 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:42 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:43.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:42 smithi114 ceph-mon[58780]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:07:43.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:42 smithi114 ceph-mon[58780]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 764 B/s rd, 0 op/s 2024-04-12T22:07:43.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:42 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:43.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:42 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:43.351 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-12T22:07:43.351 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": { 2024-04-12T22:07:43.351 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 3 2024-04-12T22:07:43.351 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:07:43.351 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": { 2024-04-12T22:07:43.351 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 2 2024-04-12T22:07:43.351 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:07:43.352 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": { 2024-04-12T22:07:43.352 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 8 2024-04-12T22:07:43.352 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:07:43.352 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw": { 2024-04-12T22:07:43.352 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-04-12T22:07:43.352 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:07:43.352 INFO:teuthology.orchestra.run.smithi097.stdout: "overall": { 2024-04-12T22:07:43.352 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-04-12T22:07:43.352 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 13 2024-04-12T22:07:43.352 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-12T22:07:43.352 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-12T22:07:43.711 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-12T22:07:43.711 INFO:teuthology.orchestra.run.smithi097.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a", 2024-04-12T22:07:43.711 INFO:teuthology.orchestra.run.smithi097.stdout: "in_progress": true, 2024-04-12T22:07:43.711 INFO:teuthology.orchestra.run.smithi097.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T22:07:43.711 INFO:teuthology.orchestra.run.smithi097.stdout: "services_complete": [ 2024-04-12T22:07:43.711 INFO:teuthology.orchestra.run.smithi097.stdout: "mon", 2024-04-12T22:07:43.711 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr", 2024-04-12T22:07:43.711 INFO:teuthology.orchestra.run.smithi097.stdout: "osd" 2024-04-12T22:07:43.711 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-04-12T22:07:43.712 INFO:teuthology.orchestra.run.smithi097.stdout: "progress": "13/23 daemons upgraded", 2024-04-12T22:07:43.712 INFO:teuthology.orchestra.run.smithi097.stdout: "message": "Currently upgrading rgw daemons", 2024-04-12T22:07:43.712 INFO:teuthology.orchestra.run.smithi097.stdout: "is_paused": false 2024-04-12T22:07:43.712 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-12T22:07:44.072 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:43 smithi097 ceph-mon[64853]: from='client.34503 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:07:44.072 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:43 smithi097 ceph-mon[64853]: from='client.34509 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:07:44.072 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:43 smithi097 ceph-mon[64853]: from='client.? 172.21.15.97:0/3805579079' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:07:44.072 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:43 smithi097 ceph-mon[67407]: from='client.34503 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:07:44.073 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:43 smithi097 ceph-mon[67407]: from='client.34509 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:07:44.073 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:43 smithi097 ceph-mon[67407]: from='client.? 172.21.15.97:0/3805579079' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:07:44.123 INFO:teuthology.orchestra.run.smithi097.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-12T22:07:44.123 INFO:teuthology.orchestra.run.smithi097.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-12T22:07:44.124 INFO:teuthology.orchestra.run.smithi097.stdout: daemon grafana.a on smithi114 is in error state 2024-04-12T22:07:44.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:43 smithi114 ceph-mon[58780]: from='client.34503 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:07:44.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:43 smithi114 ceph-mon[58780]: from='client.34509 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:07:44.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:43 smithi114 ceph-mon[58780]: from='client.? 172.21.15.97:0/3805579079' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:07:45.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:44 smithi097 ceph-mon[64853]: from='client.34539 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:07:45.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:44 smithi097 ceph-mon[64853]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-12T22:07:45.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:44 smithi097 ceph-mon[64853]: from='client.44479 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:07:45.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:44 smithi097 ceph-mon[64853]: from='client.? 172.21.15.97:0/2542530605' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T22:07:45.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:44 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:45.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:44 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi097.exxkgl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-12T22:07:45.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:44 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:07:45.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:44 smithi097 ceph-mon[67407]: from='client.34539 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:07:45.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:44 smithi097 ceph-mon[67407]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-12T22:07:45.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:44 smithi097 ceph-mon[67407]: from='client.44479 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:07:45.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:44 smithi097 ceph-mon[67407]: from='client.? 172.21.15.97:0/2542530605' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T22:07:45.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:44 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:45.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:44 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi097.exxkgl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-12T22:07:45.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:44 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:07:45.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:44 smithi114 ceph-mon[58780]: from='client.34539 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:07:45.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:44 smithi114 ceph-mon[58780]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-12T22:07:45.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:44 smithi114 ceph-mon[58780]: from='client.44479 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:07:45.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:44 smithi114 ceph-mon[58780]: from='client.? 172.21.15.97:0/2542530605' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T22:07:45.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:44 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:45.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:44 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi097.exxkgl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-12T22:07:45.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:44 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:07:45.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:45 smithi097 ceph-mon[64853]: Upgrade: Updating rgw.smpl.smithi097.exxkgl (2/4) 2024-04-12T22:07:45.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:45 smithi097 ceph-mon[64853]: Deploying daemon rgw.smpl.smithi097.exxkgl on smithi097 2024-04-12T22:07:45.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:45 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:45.910 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:45 smithi097 ceph-mon[67407]: Upgrade: Updating rgw.smpl.smithi097.exxkgl (2/4) 2024-04-12T22:07:45.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:45 smithi097 ceph-mon[67407]: Deploying daemon rgw.smpl.smithi097.exxkgl on smithi097 2024-04-12T22:07:45.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:45 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:46.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:45 smithi114 ceph-mon[58780]: Upgrade: Updating rgw.smpl.smithi097.exxkgl (2/4) 2024-04-12T22:07:46.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:45 smithi114 ceph-mon[58780]: Deploying daemon rgw.smpl.smithi097.exxkgl on smithi097 2024-04-12T22:07:46.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:45 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:47.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:46 smithi097 ceph-mon[64853]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-12T22:07:47.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:46 smithi097 ceph-mon[67407]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-12T22:07:47.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:46 smithi114 ceph-mon[58780]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-12T22:07:48.969 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:48 smithi114 ceph-mon[58780]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 94 B/s wr, 93 op/s 2024-04-12T22:07:48.970 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:48 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:48.970 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:48 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:49.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:48 smithi097 ceph-mon[64853]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 94 B/s wr, 93 op/s 2024-04-12T22:07:49.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:48 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:49.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:48 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:49.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:48 smithi097 ceph-mon[67407]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 94 B/s wr, 93 op/s 2024-04-12T22:07:49.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:48 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:49.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:48 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:49.660 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:07:49 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:07:49] "GET /metrics HTTP/1.1" 200 38637 "" "Prometheus/2.33.4" 2024-04-12T22:07:50.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:50 smithi097 ceph-mon[64853]: Upgrade: Updating rgw.foo.smithi114.ulfbyo (3/4) 2024-04-12T22:07:50.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:50 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:50.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:50 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi114.ulfbyo", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-12T22:07:50.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:50 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:07:50.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:50 smithi097 ceph-mon[64853]: Deploying daemon rgw.foo.smithi114.ulfbyo on smithi114 2024-04-12T22:07:50.911 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:50 smithi097 ceph-mon[64853]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-04-12T22:07:50.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:50 smithi097 ceph-mon[67407]: Upgrade: Updating rgw.foo.smithi114.ulfbyo (3/4) 2024-04-12T22:07:50.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:50 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:50.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:50 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi114.ulfbyo", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-12T22:07:50.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:50 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:07:50.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:50 smithi097 ceph-mon[67407]: Deploying daemon rgw.foo.smithi114.ulfbyo on smithi114 2024-04-12T22:07:50.912 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:50 smithi097 ceph-mon[67407]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-04-12T22:07:50.957 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:50 smithi114 ceph-mon[58780]: Upgrade: Updating rgw.foo.smithi114.ulfbyo (3/4) 2024-04-12T22:07:50.957 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:50 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:50.957 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:50 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi114.ulfbyo", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-12T22:07:50.957 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:50 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:07:50.958 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:50 smithi114 ceph-mon[58780]: Deploying daemon rgw.foo.smithi114.ulfbyo on smithi114 2024-04-12T22:07:50.958 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:50 smithi114 ceph-mon[58780]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-04-12T22:07:52.729 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:52 smithi114 ceph-mon[58780]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:07:52.729 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:52 smithi114 ceph-mon[58780]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 85 B/s wr, 131 op/s 2024-04-12T22:07:53.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:52 smithi097 ceph-mon[64853]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:07:53.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:52 smithi097 ceph-mon[64853]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 85 B/s wr, 131 op/s 2024-04-12T22:07:53.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:52 smithi097 ceph-mon[67407]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:07:53.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:52 smithi097 ceph-mon[67407]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 85 B/s wr, 131 op/s 2024-04-12T22:07:54.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:54 smithi097 ceph-mon[64853]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-04-12T22:07:54.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:54 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:54.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:54 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:54.910 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:54 smithi097 ceph-mon[67407]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-04-12T22:07:54.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:54 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:54.911 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:54 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:54.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:54 smithi114 ceph-mon[58780]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-04-12T22:07:54.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:54 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:54.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:54 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:55.973 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:55 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:07:55.974 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:55 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:55.974 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:55 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi114.lrgvph", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-12T22:07:55.974 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:55 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:07:56.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:55 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:07:56.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:55 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:56.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:55 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi114.lrgvph", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-12T22:07:56.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:55 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:07:56.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:55 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:07:56.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:55 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:07:56.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:55 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi114.lrgvph", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-12T22:07:56.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:55 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:07:56.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:56 smithi114 ceph-mon[58780]: Upgrade: Updating rgw.smpl.smithi114.lrgvph (4/4) 2024-04-12T22:07:56.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:56 smithi114 ceph-mon[58780]: Deploying daemon rgw.smpl.smithi114.lrgvph on smithi114 2024-04-12T22:07:56.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:56 smithi114 ceph-mon[58780]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 167 op/s 2024-04-12T22:07:57.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:56 smithi097 ceph-mon[64853]: Upgrade: Updating rgw.smpl.smithi114.lrgvph (4/4) 2024-04-12T22:07:57.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:56 smithi097 ceph-mon[64853]: Deploying daemon rgw.smpl.smithi114.lrgvph on smithi114 2024-04-12T22:07:57.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:56 smithi097 ceph-mon[64853]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 167 op/s 2024-04-12T22:07:57.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:56 smithi097 ceph-mon[67407]: Upgrade: Updating rgw.smpl.smithi114.lrgvph (4/4) 2024-04-12T22:07:57.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:56 smithi097 ceph-mon[67407]: Deploying daemon rgw.smpl.smithi114.lrgvph on smithi114 2024-04-12T22:07:57.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:56 smithi097 ceph-mon[67407]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 167 op/s 2024-04-12T22:07:58.978 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:07:58 smithi114 ceph-mon[58780]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 163 KiB/s rd, 255 B/s wr, 251 op/s 2024-04-12T22:07:59.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:07:58 smithi097 ceph-mon[64853]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 163 KiB/s rd, 255 B/s wr, 251 op/s 2024-04-12T22:07:59.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:07:58 smithi097 ceph-mon[67407]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 163 KiB/s rd, 255 B/s wr, 251 op/s 2024-04-12T22:07:59.660 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:07:59 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:07:59] "GET /metrics HTTP/1.1" 200 38626 "" "Prometheus/2.33.4" 2024-04-12T22:08:00.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:00 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:00.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:00 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:00.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:00 smithi114 ceph-mon[58780]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-04-12T22:08:00.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:00 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:00.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:00 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:00.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:00 smithi097 ceph-mon[64853]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-04-12T22:08:00.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:00 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:00.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:00 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:00.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:00 smithi097 ceph-mon[67407]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-04-12T22:08:02.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:02 smithi114 ceph-mon[58780]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:08:02.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:02 smithi114 ceph-mon[58780]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 139 KiB/s rd, 170 B/s wr, 215 op/s 2024-04-12T22:08:02.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:02 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:02.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:02 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:03.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:02 smithi097 ceph-mon[64853]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:08:03.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:02 smithi097 ceph-mon[64853]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 139 KiB/s rd, 170 B/s wr, 215 op/s 2024-04-12T22:08:03.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:02 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:03.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:02 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:03.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:02 smithi097 ceph-mon[67407]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:08:03.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:02 smithi097 ceph-mon[67407]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 139 KiB/s rd, 170 B/s wr, 215 op/s 2024-04-12T22:08:03.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:02 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:03.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:02 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:04.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:04 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:04.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:04 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:04.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:04 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:04.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:04 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:04.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:04 smithi114 ceph-mon[58780]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 134 KiB/s rd, 255 B/s wr, 206 op/s 2024-04-12T22:08:04.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:04 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:04.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:04 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:04.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:04 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:04.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:04 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:04.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:04 smithi097 ceph-mon[64853]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 134 KiB/s rd, 255 B/s wr, 206 op/s 2024-04-12T22:08:04.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:04 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:04.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:04 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:04.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:04 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:04.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:04 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:04.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:04 smithi097 ceph-mon[67407]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 134 KiB/s rd, 255 B/s wr, 206 op/s 2024-04-12T22:08:05.608 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:05 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:05.609 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:05 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:05.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:05 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:05.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:05 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:05.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:05 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:05.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:05 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:06.525 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:06 smithi097 ceph-mon[64853]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-04-12T22:08:06.525 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:06 smithi097 ceph-mon[67407]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-04-12T22:08:06.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:06 smithi114 ceph-mon[58780]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-04-12T22:08:08.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[64853]: Detected new or changed devices on smithi114 2024-04-12T22:08:08.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:08.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:08.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[64853]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 170 B/s wr, 168 op/s 2024-04-12T22:08:08.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:08.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:08.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:08:08.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:08:08.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:08.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:08:08.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:08:08.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:08:08.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:08:08.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:08:08.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:08:08.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:08:08.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:08.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi097.sggtkh"}]: dispatch 2024-04-12T22:08:08.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi097.sggtkh"}]': finished 2024-04-12T22:08:08.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi114.ulfbyo"}]: dispatch 2024-04-12T22:08:08.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi114.ulfbyo"}]': finished 2024-04-12T22:08:08.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi097.exxkgl"}]: dispatch 2024-04-12T22:08:08.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi097.exxkgl"}]': finished 2024-04-12T22:08:08.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi114.lrgvph"}]: dispatch 2024-04-12T22:08:08.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi114.lrgvph"}]': finished 2024-04-12T22:08:08.662 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:08:08.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[67407]: Detected new or changed devices on smithi114 2024-04-12T22:08:08.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:08.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:08.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[67407]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 170 B/s wr, 168 op/s 2024-04-12T22:08:08.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:08.663 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:08.663 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:08:08.663 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:08:08.663 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:08.663 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:08:08.663 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:08:08.663 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:08:08.663 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:08:08.663 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:08:08.663 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:08:08.663 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:08:08.663 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:08.663 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi097.sggtkh"}]: dispatch 2024-04-12T22:08:08.663 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi097.sggtkh"}]': finished 2024-04-12T22:08:08.663 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi114.ulfbyo"}]: dispatch 2024-04-12T22:08:08.664 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi114.ulfbyo"}]': finished 2024-04-12T22:08:08.664 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi097.exxkgl"}]: dispatch 2024-04-12T22:08:08.664 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi097.exxkgl"}]': finished 2024-04-12T22:08:08.664 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi114.lrgvph"}]: dispatch 2024-04-12T22:08:08.664 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi114.lrgvph"}]': finished 2024-04-12T22:08:08.664 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:08:08.664 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:08.664 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:08:08.664 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:08:08.664 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:08.665 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:08.665 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:08:08.665 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:08:08.665 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:08 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:08.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:08 smithi114 ceph-mon[58780]: Detected new or changed devices on smithi114 2024-04-12T22:08:08.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:08 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:08.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:08 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:08.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:08 smithi114 ceph-mon[58780]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 170 B/s wr, 168 op/s 2024-04-12T22:08:08.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:08 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:08.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:08 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:08.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:08 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:08:08.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:08 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:08:08.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:08 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:08.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:08 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:08:08.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:08 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:08:08.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:08 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:08:08.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:08 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:08:08.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:08 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:08:08.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:08 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:08:08.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:08 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:08:08.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:08 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:08.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:08 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi097.sggtkh"}]: dispatch 2024-04-12T22:08:08.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:08 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi097.sggtkh"}]': finished 2024-04-12T22:08:08.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:08 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi114.ulfbyo"}]: dispatch 2024-04-12T22:08:08.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:08 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi114.ulfbyo"}]': finished 2024-04-12T22:08:08.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:08 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi097.exxkgl"}]: dispatch 2024-04-12T22:08:08.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:08 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi097.exxkgl"}]': finished 2024-04-12T22:08:08.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:08 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi114.lrgvph"}]: dispatch 2024-04-12T22:08:08.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:08 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi114.lrgvph"}]': finished 2024-04-12T22:08:08.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:08 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:08:08.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:08 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:08.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:08 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:08:08.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:08 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:08:08.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:08 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:09.321 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:08:09 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:08:09] "GET /metrics HTTP/1.1" 200 38626 "" "Prometheus/2.33.4" 2024-04-12T22:08:09.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:09 smithi097 ceph-mon[64853]: Detected new or changed devices on smithi097 2024-04-12T22:08:09.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:09 smithi097 ceph-mon[64853]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 99 B/s wr, 98 op/s 2024-04-12T22:08:09.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:09 smithi097 ceph-mon[64853]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 123 B/s wr, 122 op/s 2024-04-12T22:08:09.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:09 smithi097 ceph-mon[64853]: Upgrade: Setting container_image for all rgw 2024-04-12T22:08:09.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:09 smithi097 ceph-mon[64853]: Upgrade: Setting container_image for all rbd-mirror 2024-04-12T22:08:09.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:09 smithi097 ceph-mon[64853]: Upgrade: Setting container_image for all ceph-exporter 2024-04-12T22:08:09.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:09 smithi097 ceph-mon[67407]: Detected new or changed devices on smithi097 2024-04-12T22:08:09.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:09 smithi097 ceph-mon[67407]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 99 B/s wr, 98 op/s 2024-04-12T22:08:09.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:09 smithi097 ceph-mon[67407]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 123 B/s wr, 122 op/s 2024-04-12T22:08:09.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:09 smithi097 ceph-mon[67407]: Upgrade: Setting container_image for all rgw 2024-04-12T22:08:09.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:09 smithi097 ceph-mon[67407]: Upgrade: Setting container_image for all rbd-mirror 2024-04-12T22:08:09.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:09 smithi097 ceph-mon[67407]: Upgrade: Setting container_image for all ceph-exporter 2024-04-12T22:08:09.742 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:09 smithi114 ceph-mon[58780]: Detected new or changed devices on smithi097 2024-04-12T22:08:09.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:09 smithi114 ceph-mon[58780]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 99 B/s wr, 98 op/s 2024-04-12T22:08:09.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:09 smithi114 ceph-mon[58780]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 123 B/s wr, 122 op/s 2024-04-12T22:08:09.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:09 smithi114 ceph-mon[58780]: Upgrade: Setting container_image for all rgw 2024-04-12T22:08:09.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:09 smithi114 ceph-mon[58780]: Upgrade: Setting container_image for all rbd-mirror 2024-04-12T22:08:09.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:09 smithi114 ceph-mon[58780]: Upgrade: Setting container_image for all ceph-exporter 2024-04-12T22:08:10.624 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:10 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:08:10.624 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:10 smithi097 ceph-mon[64853]: Upgrade: Updating iscsi.foo.smithi097.xndkxy 2024-04-12T22:08:10.624 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:10 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:10.624 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:10 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi097.xndkxy", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-12T22:08:10.624 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:10 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:08:10.624 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:10 smithi097 ceph-mon[64853]: Deploying daemon iscsi.foo.smithi097.xndkxy on smithi097 2024-04-12T22:08:10.624 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:10 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:08:10.625 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:10 smithi097 ceph-mon[67407]: Upgrade: Updating iscsi.foo.smithi097.xndkxy 2024-04-12T22:08:10.625 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:10 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:10.625 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:10 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi097.xndkxy", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-12T22:08:10.625 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:10 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:08:10.625 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:10 smithi097 ceph-mon[67407]: Deploying daemon iscsi.foo.smithi097.xndkxy on smithi097 2024-04-12T22:08:10.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:10 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:08:10.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:10 smithi114 ceph-mon[58780]: Upgrade: Updating iscsi.foo.smithi097.xndkxy 2024-04-12T22:08:10.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:10 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:10.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:10 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi097.xndkxy", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-12T22:08:10.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:10 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:08:10.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:10 smithi114 ceph-mon[58780]: Deploying daemon iscsi.foo.smithi097.xndkxy on smithi097 2024-04-12T22:08:11.942 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:11 smithi097 ceph-mon[64853]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 123 B/s wr, 54 op/s 2024-04-12T22:08:11.942 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:11 smithi097 ceph-mon[67407]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 123 B/s wr, 54 op/s 2024-04-12T22:08:12.242 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:11 smithi114 ceph-mon[58780]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 123 B/s wr, 54 op/s 2024-04-12T22:08:13.051 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:12 smithi097 ceph-mon[64853]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:08:13.052 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:12 smithi097 ceph-mon[67407]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:08:13.242 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:12 smithi114 ceph-mon[58780]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:08:14.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:13 smithi097 ceph-mon[64853]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:08:14.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:13 smithi097 ceph-mon[67407]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:08:14.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:13 smithi114 ceph-mon[58780]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:08:14.474 INFO:teuthology.orchestra.run.smithi097.stdout:true 2024-04-12T22:08:15.123 INFO:teuthology.orchestra.run.smithi097.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T22:08:15.123 INFO:teuthology.orchestra.run.smithi097.stdout:alertmanager.a smithi097 *:9093,9094 running (7m) 11s ago 12m 24.0M - 0.23.0 ba2b418f427c 7b4c491fe0a0 2024-04-12T22:08:15.123 INFO:teuthology.orchestra.run.smithi097.stdout:grafana.a smithi114 *:3000 error 12s ago 11m - - 2024-04-12T22:08:15.123 INFO:teuthology.orchestra.run.smithi097.stdout:iscsi.foo.smithi097.xndkxy smithi097 running (6m) 11s ago 10m 56.8M - 3.5 e1d6a67b021e 0436463f28cd 2024-04-12T22:08:15.123 INFO:teuthology.orchestra.run.smithi097.stdout:mgr.x smithi114 *:8443,9283,8765 running (6m) 12s ago 16m 452M - 19.0.0-2053-gb277feb2 668895944f0e b8f8df3bf928 2024-04-12T22:08:15.123 INFO:teuthology.orchestra.run.smithi097.stdout:mgr.y smithi097 *:8443,9283,8765 running (6m) 11s ago 18m 513M - 19.0.0-2053-gb277feb2 668895944f0e 488170dac11f 2024-04-12T22:08:15.123 INFO:teuthology.orchestra.run.smithi097.stdout:mon.a smithi097 running (5m) 11s ago 18m 59.7M 2048M 19.0.0-2053-gb277feb2 668895944f0e 3cfa16d3a85c 2024-04-12T22:08:15.124 INFO:teuthology.orchestra.run.smithi097.stdout:mon.b smithi114 running (5m) 12s ago 16m 43.3M 2048M 19.0.0-2053-gb277feb2 668895944f0e 71a2eff159a3 2024-04-12T22:08:15.124 INFO:teuthology.orchestra.run.smithi097.stdout:mon.c smithi097 running (5m) 11s ago 16m 46.8M 2048M 19.0.0-2053-gb277feb2 668895944f0e 69507ba88603 2024-04-12T22:08:15.124 INFO:teuthology.orchestra.run.smithi097.stdout:node-exporter.a smithi097 *:9100 running (7m) 11s ago 12m 21.2M - 1.3.1 1dbe0e931976 5edff9053b37 2024-04-12T22:08:15.124 INFO:teuthology.orchestra.run.smithi097.stdout:node-exporter.b smithi114 *:9100 running (6m) 12s ago 12m 17.7M - 1.3.1 1dbe0e931976 904afc45dc2f 2024-04-12T22:08:15.124 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0 smithi097 running (4m) 11s ago 16m 63.8M 2736M 19.0.0-2053-gb277feb2 668895944f0e 9d05fea4bf37 2024-04-12T22:08:15.124 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1 smithi097 running (3m) 11s ago 15m 56.2M 2736M 19.0.0-2053-gb277feb2 668895944f0e a9a527642889 2024-04-12T22:08:15.124 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2 smithi097 running (3m) 11s ago 15m 52.5M 2736M 19.0.0-2053-gb277feb2 668895944f0e 2e2297afff50 2024-04-12T22:08:15.124 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3 smithi097 running (2m) 11s ago 15m 62.1M 2736M 19.0.0-2053-gb277feb2 668895944f0e 415b4d3d85f2 2024-04-12T22:08:15.124 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4 smithi114 running (2m) 12s ago 14m 56.2M 2992M 19.0.0-2053-gb277feb2 668895944f0e fcd2e39a5866 2024-04-12T22:08:15.124 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5 smithi114 running (117s) 12s ago 14m 60.3M 2992M 19.0.0-2053-gb277feb2 668895944f0e 470c0780e428 2024-04-12T22:08:15.124 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6 smithi114 running (83s) 12s ago 13m 59.4M 2992M 19.0.0-2053-gb277feb2 668895944f0e 6b80aaeca08b 2024-04-12T22:08:15.124 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7 smithi114 running (48s) 12s ago 13m 56.2M 2992M 19.0.0-2053-gb277feb2 668895944f0e 1c3e102eaf74 2024-04-12T22:08:15.124 INFO:teuthology.orchestra.run.smithi097.stdout:prometheus.a smithi114 *:9095 running (6m) 12s ago 12m 71.9M - 2.33.4 514e6a882f6e 0ad5437ed63f 2024-04-12T22:08:15.124 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.foo.smithi097.sggtkh smithi097 *:8000 running (33s) 11s ago 11m 76.7M - 19.0.0-2053-gb277feb2 668895944f0e 96fa214b3dd8 2024-04-12T22:08:15.124 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.foo.smithi114.ulfbyo smithi114 *:8000 running (21s) 12s ago 11m 75.1M - 19.0.0-2053-gb277feb2 668895944f0e 02e33119b3ca 2024-04-12T22:08:15.125 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.smpl.smithi097.exxkgl smithi097 *:80 running (27s) 11s ago 11m 77.3M - 19.0.0-2053-gb277feb2 668895944f0e 7d210f6a20c9 2024-04-12T22:08:15.125 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.smpl.smithi114.lrgvph smithi114 *:80 running (16s) 12s ago 11m 74.7M - 19.0.0-2053-gb277feb2 668895944f0e 646ff8b6fed5 2024-04-12T22:08:15.520 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-12T22:08:15.520 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": { 2024-04-12T22:08:15.521 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 3 2024-04-12T22:08:15.521 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:08:15.521 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": { 2024-04-12T22:08:15.521 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 2 2024-04-12T22:08:15.521 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:08:15.521 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": { 2024-04-12T22:08:15.521 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 8 2024-04-12T22:08:15.521 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:08:15.521 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw": { 2024-04-12T22:08:15.521 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 4 2024-04-12T22:08:15.521 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:08:15.521 INFO:teuthology.orchestra.run.smithi097.stdout: "overall": { 2024-04-12T22:08:15.521 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 17 2024-04-12T22:08:15.521 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-12T22:08:15.521 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-12T22:08:15.848 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-12T22:08:15.848 INFO:teuthology.orchestra.run.smithi097.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a", 2024-04-12T22:08:15.849 INFO:teuthology.orchestra.run.smithi097.stdout: "in_progress": true, 2024-04-12T22:08:15.849 INFO:teuthology.orchestra.run.smithi097.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T22:08:15.849 INFO:teuthology.orchestra.run.smithi097.stdout: "services_complete": [ 2024-04-12T22:08:15.849 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw", 2024-04-12T22:08:15.849 INFO:teuthology.orchestra.run.smithi097.stdout: "mon", 2024-04-12T22:08:15.849 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr", 2024-04-12T22:08:15.849 INFO:teuthology.orchestra.run.smithi097.stdout: "osd" 2024-04-12T22:08:15.849 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-04-12T22:08:15.849 INFO:teuthology.orchestra.run.smithi097.stdout: "progress": "17/23 daemons upgraded", 2024-04-12T22:08:15.849 INFO:teuthology.orchestra.run.smithi097.stdout: "message": "Currently upgrading iscsi daemons", 2024-04-12T22:08:15.849 INFO:teuthology.orchestra.run.smithi097.stdout: "is_paused": false 2024-04-12T22:08:15.849 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-12T22:08:16.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:15 smithi097 ceph-mon[67407]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:08:16.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:15 smithi097 ceph-mon[67407]: from='client.34620 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:08:16.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:15 smithi097 ceph-mon[67407]: from='client.34626 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:08:16.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:15 smithi097 ceph-mon[67407]: from='client.? 172.21.15.97:0/2729036766' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:08:16.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:15 smithi097 ceph-mon[64853]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:08:16.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:15 smithi097 ceph-mon[64853]: from='client.34620 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:08:16.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:15 smithi097 ceph-mon[64853]: from='client.34626 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:08:16.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:15 smithi097 ceph-mon[64853]: from='client.? 172.21.15.97:0/2729036766' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:08:16.243 INFO:teuthology.orchestra.run.smithi097.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-12T22:08:16.243 INFO:teuthology.orchestra.run.smithi097.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-12T22:08:16.243 INFO:teuthology.orchestra.run.smithi097.stdout: daemon grafana.a on smithi114 is in error state 2024-04-12T22:08:16.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:15 smithi114 ceph-mon[58780]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:08:16.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:15 smithi114 ceph-mon[58780]: from='client.34620 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:08:16.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:15 smithi114 ceph-mon[58780]: from='client.34626 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:08:16.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:15 smithi114 ceph-mon[58780]: from='client.? 172.21.15.97:0/2729036766' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:08:17.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:16 smithi097 ceph-mon[64853]: from='client.44563 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:08:17.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:16 smithi097 ceph-mon[64853]: from='client.34644 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:08:17.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:16 smithi097 ceph-mon[64853]: from='client.? 172.21.15.97:0/3732846233' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T22:08:17.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:16 smithi097 ceph-mon[67407]: from='client.44563 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:08:17.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:16 smithi097 ceph-mon[67407]: from='client.34644 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:08:17.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:16 smithi097 ceph-mon[67407]: from='client.? 172.21.15.97:0/3732846233' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T22:08:17.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:16 smithi114 ceph-mon[58780]: from='client.44563 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:08:17.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:16 smithi114 ceph-mon[58780]: from='client.34644 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:08:17.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:16 smithi114 ceph-mon[58780]: from='client.? 172.21.15.97:0/3732846233' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T22:08:18.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:17 smithi097 ceph-mon[64853]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 615 B/s rd, 0 op/s 2024-04-12T22:08:18.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:17 smithi097 ceph-mon[67407]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 615 B/s rd, 0 op/s 2024-04-12T22:08:18.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:17 smithi114 ceph-mon[58780]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 615 B/s rd, 0 op/s 2024-04-12T22:08:19.660 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:08:19 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:08:19] "GET /metrics HTTP/1.1" 200 38694 "" "Prometheus/2.33.4" 2024-04-12T22:08:20.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:19 smithi097 ceph-mon[64853]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-12T22:08:20.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:19 smithi097 ceph-mon[67407]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-12T22:08:20.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:19 smithi114 ceph-mon[58780]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-12T22:08:22.159 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:21 smithi097 ceph-mon[64853]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:08:22.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:21 smithi097 ceph-mon[67407]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:08:22.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:21 smithi114 ceph-mon[58780]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:08:23.016 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:22 smithi097 ceph-mon[64853]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:08:23.017 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:22 smithi097 ceph-mon[67407]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:08:23.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:22 smithi114 ceph-mon[58780]: from='client.25315 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:08:23.660 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:08:23 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:08:23.341+0000 7fe3e01f2700 -1 log_channel(cephadm) log [ERR] : Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi097.xndkxy on host smithi097 failed. 2024-04-12T22:08:24.007 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:23 smithi097 ceph-mon[64853]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:08:24.007 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:23 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:24.008 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:23 smithi097 ceph-mon[67407]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:08:24.008 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:23 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:24.139 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:23 smithi114 ceph-mon[58780]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:08:24.139 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:23 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:25.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:24 smithi097 ceph-mon[64853]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi097.xndkxy on host smithi097 failed. 2024-04-12T22:08:25.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:24 smithi097 ceph-mon[64853]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 900 B/s rd, 0 op/s 2024-04-12T22:08:25.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:24 smithi097 ceph-mon[64853]: Health check failed: Upgrading daemon iscsi.foo.smithi097.xndkxy on host smithi097 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-04-12T22:08:25.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:24 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:25.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:24 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:08:25.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:24 smithi097 ceph-mon[67407]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi097.xndkxy on host smithi097 failed. 2024-04-12T22:08:25.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:24 smithi097 ceph-mon[67407]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 900 B/s rd, 0 op/s 2024-04-12T22:08:25.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:24 smithi097 ceph-mon[67407]: Health check failed: Upgrading daemon iscsi.foo.smithi097.xndkxy on host smithi097 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-04-12T22:08:25.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:24 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:25.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:24 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:08:25.178 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:24 smithi114 ceph-mon[58780]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi097.xndkxy on host smithi097 failed. 2024-04-12T22:08:25.178 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:24 smithi114 ceph-mon[58780]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 900 B/s rd, 0 op/s 2024-04-12T22:08:25.178 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:24 smithi114 ceph-mon[58780]: Health check failed: Upgrading daemon iscsi.foo.smithi097.xndkxy on host smithi097 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-04-12T22:08:25.178 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:24 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:25.178 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:24 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:08:27.131 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:26 smithi114 ceph-mon[58780]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 900 B/s rd, 0 op/s 2024-04-12T22:08:27.131 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:26 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:27.131 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:26 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:27.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:26 smithi097 ceph-mon[64853]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 900 B/s rd, 0 op/s 2024-04-12T22:08:27.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:26 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:27.161 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:26 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:27.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:26 smithi097 ceph-mon[67407]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 900 B/s rd, 0 op/s 2024-04-12T22:08:27.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:26 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:27.161 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:26 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:28.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:28 smithi097 ceph-mon[64853]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 900 B/s rd, 0 op/s 2024-04-12T22:08:28.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:28 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:28.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:28 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:28.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:28 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:08:28.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:28 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:08:28.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:28 smithi097 ceph-mon[64853]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 544 B/s rd, 0 op/s 2024-04-12T22:08:28.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:28 smithi097 ceph-mon[64853]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 690 B/s rd, 0 op/s 2024-04-12T22:08:28.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:28 smithi097 ceph-mon[64853]: Checking dashboard <-> RGW credentials 2024-04-12T22:08:28.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:28 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:28.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:28 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:08:28.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:28 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:08:28.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:28 smithi097 ceph-mon[64853]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:08:28.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:28 smithi097 ceph-mon[64853]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:08:28.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:28 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:28.661 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:28 smithi097 ceph-mon[64853]: Health check update: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-12T22:08:28.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:28 smithi097 ceph-mon[67407]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 900 B/s rd, 0 op/s 2024-04-12T22:08:28.661 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:28 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:28.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:28 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:28.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:28 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:08:28.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:28 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:08:28.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:28 smithi097 ceph-mon[67407]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 544 B/s rd, 0 op/s 2024-04-12T22:08:28.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:28 smithi097 ceph-mon[67407]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 690 B/s rd, 0 op/s 2024-04-12T22:08:28.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:28 smithi097 ceph-mon[67407]: Checking dashboard <-> RGW credentials 2024-04-12T22:08:28.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:28 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:28.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:28 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:08:28.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:28 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:08:28.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:28 smithi097 ceph-mon[67407]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:08:28.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:28 smithi097 ceph-mon[67407]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:08:28.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:28 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:28.662 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:28 smithi097 ceph-mon[67407]: Health check update: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-12T22:08:28.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:28 smithi114 ceph-mon[58780]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 900 B/s rd, 0 op/s 2024-04-12T22:08:28.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:28 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:28.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:28 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:28.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:28 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:08:28.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:28 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:08:28.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:28 smithi114 ceph-mon[58780]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 544 B/s rd, 0 op/s 2024-04-12T22:08:28.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:28 smithi114 ceph-mon[58780]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 690 B/s rd, 0 op/s 2024-04-12T22:08:28.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:28 smithi114 ceph-mon[58780]: Checking dashboard <-> RGW credentials 2024-04-12T22:08:28.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:28 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:28.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:28 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:08:28.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:28 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:08:28.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:28 smithi114 ceph-mon[58780]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:08:28.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:28 smithi114 ceph-mon[58780]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:08:28.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:28 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:08:28.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:28 smithi114 ceph-mon[58780]: Health check update: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-12T22:08:29.660 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:08:29 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:08:29] "GET /metrics HTTP/1.1" 200 38774 "" "Prometheus/2.33.4" 2024-04-12T22:08:31.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:30 smithi097 ceph-mon[64853]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:08:31.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:30 smithi097 ceph-mon[67407]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:08:31.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:30 smithi114 ceph-mon[58780]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:08:33.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:32 smithi097 ceph-mon[64853]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 3 op/s 2024-04-12T22:08:33.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:32 smithi097 ceph-mon[67407]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 3 op/s 2024-04-12T22:08:33.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:32 smithi114 ceph-mon[58780]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 3 op/s 2024-04-12T22:08:34.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:34 smithi097 ceph-mon[64853]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 3 op/s 2024-04-12T22:08:34.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:34 smithi097 ceph-mon[64853]: from='client.? 172.21.15.97:0/2858081334' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-12T22:08:34.910 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:34 smithi097 ceph-mon[67407]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 3 op/s 2024-04-12T22:08:34.910 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:34 smithi097 ceph-mon[67407]: from='client.? 172.21.15.97:0/2858081334' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-12T22:08:35.242 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:34 smithi114 ceph-mon[58780]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 3 op/s 2024-04-12T22:08:35.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:34 smithi114 ceph-mon[58780]: from='client.? 172.21.15.97:0/2858081334' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-12T22:08:36.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:35 smithi097 ceph-mon[64853]: from='client.? 172.21.15.97:0/1000224173' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6801/1443857790"}]: dispatch 2024-04-12T22:08:36.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:35 smithi097 ceph-mon[64853]: from='client.? ' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6801/1443857790"}]: dispatch 2024-04-12T22:08:36.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:35 smithi097 ceph-mon[67407]: from='client.? 172.21.15.97:0/1000224173' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6801/1443857790"}]: dispatch 2024-04-12T22:08:36.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:35 smithi097 ceph-mon[67407]: from='client.? ' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6801/1443857790"}]: dispatch 2024-04-12T22:08:36.242 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:35 smithi114 ceph-mon[58780]: from='client.? 172.21.15.97:0/1000224173' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6801/1443857790"}]: dispatch 2024-04-12T22:08:36.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:35 smithi114 ceph-mon[58780]: from='client.? ' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6801/1443857790"}]: dispatch 2024-04-12T22:08:37.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:36 smithi097 ceph-mon[67407]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 2 op/s 2024-04-12T22:08:37.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:36 smithi097 ceph-mon[67407]: from='client.? ' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6801/1443857790"}]': finished 2024-04-12T22:08:37.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:36 smithi097 ceph-mon[67407]: osdmap e154: 8 total, 8 up, 8 in 2024-04-12T22:08:37.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:36 smithi097 ceph-mon[67407]: from='client.? 172.21.15.97:0/2091526619' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6800/1443857790"}]: dispatch 2024-04-12T22:08:37.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:36 smithi097 ceph-mon[64853]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 2 op/s 2024-04-12T22:08:37.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:36 smithi097 ceph-mon[64853]: from='client.? ' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6801/1443857790"}]': finished 2024-04-12T22:08:37.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:36 smithi097 ceph-mon[64853]: osdmap e154: 8 total, 8 up, 8 in 2024-04-12T22:08:37.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:36 smithi097 ceph-mon[64853]: from='client.? 172.21.15.97:0/2091526619' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6800/1443857790"}]: dispatch 2024-04-12T22:08:37.242 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:36 smithi114 ceph-mon[58780]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 2 op/s 2024-04-12T22:08:37.242 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:36 smithi114 ceph-mon[58780]: from='client.? ' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6801/1443857790"}]': finished 2024-04-12T22:08:37.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:36 smithi114 ceph-mon[58780]: osdmap e154: 8 total, 8 up, 8 in 2024-04-12T22:08:37.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:36 smithi114 ceph-mon[58780]: from='client.? 172.21.15.97:0/2091526619' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6800/1443857790"}]: dispatch 2024-04-12T22:08:38.125 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:37 smithi114 ceph-mon[58780]: from='client.? 172.21.15.97:0/2091526619' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6800/1443857790"}]': finished 2024-04-12T22:08:38.125 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:37 smithi114 ceph-mon[58780]: osdmap e155: 8 total, 8 up, 8 in 2024-04-12T22:08:38.125 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:37 smithi114 ceph-mon[58780]: from='client.? 172.21.15.97:0/3271398027' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/591336683"}]: dispatch 2024-04-12T22:08:38.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:37 smithi097 ceph-mon[64853]: from='client.? 172.21.15.97:0/2091526619' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6800/1443857790"}]': finished 2024-04-12T22:08:38.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:37 smithi097 ceph-mon[64853]: osdmap e155: 8 total, 8 up, 8 in 2024-04-12T22:08:38.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:37 smithi097 ceph-mon[64853]: from='client.? 172.21.15.97:0/3271398027' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/591336683"}]: dispatch 2024-04-12T22:08:38.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:37 smithi097 ceph-mon[67407]: from='client.? 172.21.15.97:0/2091526619' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:6800/1443857790"}]': finished 2024-04-12T22:08:38.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:37 smithi097 ceph-mon[67407]: osdmap e155: 8 total, 8 up, 8 in 2024-04-12T22:08:38.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:37 smithi097 ceph-mon[67407]: from='client.? 172.21.15.97:0/3271398027' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/591336683"}]: dispatch 2024-04-12T22:08:39.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:38 smithi097 ceph-mon[64853]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 3 op/s 2024-04-12T22:08:39.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:38 smithi097 ceph-mon[64853]: from='client.? 172.21.15.97:0/3271398027' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/591336683"}]': finished 2024-04-12T22:08:39.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:38 smithi097 ceph-mon[64853]: osdmap e156: 8 total, 8 up, 8 in 2024-04-12T22:08:39.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:38 smithi097 ceph-mon[64853]: from='client.? 172.21.15.97:0/2021913959' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/4205555848"}]: dispatch 2024-04-12T22:08:39.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:38 smithi097 ceph-mon[64853]: from='client.? ' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/4205555848"}]: dispatch 2024-04-12T22:08:39.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:38 smithi097 ceph-mon[67407]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 3 op/s 2024-04-12T22:08:39.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:38 smithi097 ceph-mon[67407]: from='client.? 172.21.15.97:0/3271398027' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/591336683"}]': finished 2024-04-12T22:08:39.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:38 smithi097 ceph-mon[67407]: osdmap e156: 8 total, 8 up, 8 in 2024-04-12T22:08:39.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:38 smithi097 ceph-mon[67407]: from='client.? 172.21.15.97:0/2021913959' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/4205555848"}]: dispatch 2024-04-12T22:08:39.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:38 smithi097 ceph-mon[67407]: from='client.? ' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/4205555848"}]: dispatch 2024-04-12T22:08:39.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:38 smithi114 ceph-mon[58780]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 3 op/s 2024-04-12T22:08:39.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:38 smithi114 ceph-mon[58780]: from='client.? 172.21.15.97:0/3271398027' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/591336683"}]': finished 2024-04-12T22:08:39.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:38 smithi114 ceph-mon[58780]: osdmap e156: 8 total, 8 up, 8 in 2024-04-12T22:08:39.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:38 smithi114 ceph-mon[58780]: from='client.? 172.21.15.97:0/2021913959' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/4205555848"}]: dispatch 2024-04-12T22:08:39.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:38 smithi114 ceph-mon[58780]: from='client.? ' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/4205555848"}]: dispatch 2024-04-12T22:08:39.660 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:08:39 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:08:39] "GET /metrics HTTP/1.1" 200 38774 "" "Prometheus/2.33.4" 2024-04-12T22:08:40.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:39 smithi097 ceph-mon[64853]: from='client.? ' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/4205555848"}]': finished 2024-04-12T22:08:40.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:39 smithi097 ceph-mon[64853]: osdmap e157: 8 total, 8 up, 8 in 2024-04-12T22:08:40.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:39 smithi097 ceph-mon[64853]: from='client.? 172.21.15.97:0/3864685909' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/144405635"}]: dispatch 2024-04-12T22:08:40.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:39 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:08:40.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:39 smithi097 ceph-mon[67407]: from='client.? ' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/4205555848"}]': finished 2024-04-12T22:08:40.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:39 smithi097 ceph-mon[67407]: osdmap e157: 8 total, 8 up, 8 in 2024-04-12T22:08:40.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:39 smithi097 ceph-mon[67407]: from='client.? 172.21.15.97:0/3864685909' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/144405635"}]: dispatch 2024-04-12T22:08:40.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:39 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:08:40.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:39 smithi114 ceph-mon[58780]: from='client.? ' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/4205555848"}]': finished 2024-04-12T22:08:40.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:39 smithi114 ceph-mon[58780]: osdmap e157: 8 total, 8 up, 8 in 2024-04-12T22:08:40.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:39 smithi114 ceph-mon[58780]: from='client.? 172.21.15.97:0/3864685909' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/144405635"}]: dispatch 2024-04-12T22:08:40.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:39 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:08:41.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:40 smithi097 ceph-mon[64853]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-12T22:08:41.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:40 smithi097 ceph-mon[64853]: from='client.? 172.21.15.97:0/3864685909' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/144405635"}]': finished 2024-04-12T22:08:41.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:40 smithi097 ceph-mon[64853]: osdmap e158: 8 total, 8 up, 8 in 2024-04-12T22:08:41.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:40 smithi097 ceph-mon[64853]: from='client.? 172.21.15.97:0/3026465682' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/274023229"}]: dispatch 2024-04-12T22:08:41.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:40 smithi097 ceph-mon[67407]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-12T22:08:41.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:40 smithi097 ceph-mon[67407]: from='client.? 172.21.15.97:0/3864685909' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/144405635"}]': finished 2024-04-12T22:08:41.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:40 smithi097 ceph-mon[67407]: osdmap e158: 8 total, 8 up, 8 in 2024-04-12T22:08:41.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:40 smithi097 ceph-mon[67407]: from='client.? 172.21.15.97:0/3026465682' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/274023229"}]: dispatch 2024-04-12T22:08:41.242 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:40 smithi114 ceph-mon[58780]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-12T22:08:41.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:40 smithi114 ceph-mon[58780]: from='client.? 172.21.15.97:0/3864685909' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/144405635"}]': finished 2024-04-12T22:08:41.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:40 smithi114 ceph-mon[58780]: osdmap e158: 8 total, 8 up, 8 in 2024-04-12T22:08:41.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:40 smithi114 ceph-mon[58780]: from='client.? 172.21.15.97:0/3026465682' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/274023229"}]: dispatch 2024-04-12T22:08:42.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:41 smithi097 ceph-mon[67407]: from='client.? 172.21.15.97:0/3026465682' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/274023229"}]': finished 2024-04-12T22:08:42.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:41 smithi097 ceph-mon[67407]: osdmap e159: 8 total, 8 up, 8 in 2024-04-12T22:08:42.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:41 smithi097 ceph-mon[64853]: from='client.? 172.21.15.97:0/3026465682' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/274023229"}]': finished 2024-04-12T22:08:42.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:41 smithi097 ceph-mon[64853]: osdmap e159: 8 total, 8 up, 8 in 2024-04-12T22:08:42.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:41 smithi114 ceph-mon[58780]: from='client.? 172.21.15.97:0/3026465682' entity='client.iscsi.foo.smithi097.xndkxy' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.97:0/274023229"}]': finished 2024-04-12T22:08:42.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:41 smithi114 ceph-mon[58780]: osdmap e159: 8 total, 8 up, 8 in 2024-04-12T22:08:43.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:42 smithi097 ceph-mon[67407]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:08:43.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:42 smithi097 ceph-mon[64853]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:08:43.242 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:42 smithi114 ceph-mon[58780]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail 2024-04-12T22:08:45.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:44 smithi097 ceph-mon[67407]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 864 B/s rd, 0 op/s 2024-04-12T22:08:45.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:44 smithi097 ceph-mon[64853]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 864 B/s rd, 0 op/s 2024-04-12T22:08:45.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:44 smithi114 ceph-mon[58780]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 864 B/s rd, 0 op/s 2024-04-12T22:08:45.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:45 smithi114 ceph-mon[58780]: from='client.54473 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:08:46.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:45 smithi097 ceph-mon[64853]: from='client.54473 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:08:46.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:45 smithi097 ceph-mon[67407]: from='client.54473 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:08:46.598 INFO:teuthology.orchestra.run.smithi097.stdout:true 2024-04-12T22:08:46.909 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:46 smithi097 ceph-mon[64853]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 739 B/s rd, 0 op/s 2024-04-12T22:08:46.910 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:46 smithi097 ceph-mon[67407]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 739 B/s rd, 0 op/s 2024-04-12T22:08:46.931 INFO:teuthology.orchestra.run.smithi097.stdout:"Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi097.xndkxy on host smithi097 failed." 2024-04-12T22:08:47.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:46 smithi114 ceph-mon[58780]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 739 B/s rd, 0 op/s 2024-04-12T22:08:47.741 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -e sha1=b277feb2c7beb4597d8ecc39d58d5c36b81f613a -- bash -c 'ceph orch ps' 2024-04-12T22:08:48.053 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:47 smithi097 ceph-mon[64853]: from='client.44683 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:08:48.054 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:47 smithi097 ceph-mon[67407]: from='client.44683 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:08:48.242 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:47 smithi114 ceph-mon[58780]: from='client.44683 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:08:49.064 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:48 smithi097 ceph-mon[64853]: from='client.44689 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:08:49.065 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:48 smithi097 ceph-mon[64853]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:08:49.065 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:48 smithi097 ceph-mon[67407]: from='client.44689 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:08:49.065 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:48 smithi097 ceph-mon[67407]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:08:49.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:48 smithi114 ceph-mon[58780]: from='client.44689 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:08:49.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:48 smithi114 ceph-mon[58780]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:08:49.318 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:08:49 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:08:49] "GET /metrics HTTP/1.1" 200 38774 "" "Prometheus/2.33.4" 2024-04-12T22:08:49.753 INFO:teuthology.orchestra.run.smithi097.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T22:08:49.753 INFO:teuthology.orchestra.run.smithi097.stdout:alertmanager.a smithi097 *:9093,9094 running (7m) 22s ago 12m 24.1M - 0.23.0 ba2b418f427c 7b4c491fe0a0 2024-04-12T22:08:49.753 INFO:teuthology.orchestra.run.smithi097.stdout:grafana.a smithi114 *:3000 error 23s ago 12m - - 2024-04-12T22:08:49.753 INFO:teuthology.orchestra.run.smithi097.stdout:iscsi.foo.smithi097.xndkxy smithi097 unknown 22s ago 11m - - 2024-04-12T22:08:49.753 INFO:teuthology.orchestra.run.smithi097.stdout:mgr.x smithi114 *:8443,9283,8765 running (6m) 23s ago 17m 452M - 19.0.0-2053-gb277feb2 668895944f0e b8f8df3bf928 2024-04-12T22:08:49.753 INFO:teuthology.orchestra.run.smithi097.stdout:mgr.y smithi097 *:8443,9283,8765 running (7m) 22s ago 18m 514M - 19.0.0-2053-gb277feb2 668895944f0e 488170dac11f 2024-04-12T22:08:49.753 INFO:teuthology.orchestra.run.smithi097.stdout:mon.a smithi097 running (6m) 22s ago 19m 60.9M 2048M 19.0.0-2053-gb277feb2 668895944f0e 3cfa16d3a85c 2024-04-12T22:08:49.753 INFO:teuthology.orchestra.run.smithi097.stdout:mon.b smithi114 running (5m) 23s ago 17m 44.5M 2048M 19.0.0-2053-gb277feb2 668895944f0e 71a2eff159a3 2024-04-12T22:08:49.753 INFO:teuthology.orchestra.run.smithi097.stdout:mon.c smithi097 running (5m) 22s ago 17m 48.0M 2048M 19.0.0-2053-gb277feb2 668895944f0e 69507ba88603 2024-04-12T22:08:49.753 INFO:teuthology.orchestra.run.smithi097.stdout:node-exporter.a smithi097 *:9100 running (7m) 22s ago 13m 18.4M - 1.3.1 1dbe0e931976 5edff9053b37 2024-04-12T22:08:49.753 INFO:teuthology.orchestra.run.smithi097.stdout:node-exporter.b smithi114 *:9100 running (7m) 23s ago 13m 17.1M - 1.3.1 1dbe0e931976 904afc45dc2f 2024-04-12T22:08:49.754 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0 smithi097 running (4m) 22s ago 16m 64.0M 2736M 19.0.0-2053-gb277feb2 668895944f0e 9d05fea4bf37 2024-04-12T22:08:49.754 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1 smithi097 running (4m) 22s ago 16m 56.5M 2736M 19.0.0-2053-gb277feb2 668895944f0e a9a527642889 2024-04-12T22:08:49.754 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2 smithi097 running (3m) 22s ago 16m 52.7M 2736M 19.0.0-2053-gb277feb2 668895944f0e 2e2297afff50 2024-04-12T22:08:49.754 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3 smithi097 running (3m) 22s ago 15m 62.3M 2736M 19.0.0-2053-gb277feb2 668895944f0e 415b4d3d85f2 2024-04-12T22:08:49.754 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4 smithi114 running (3m) 23s ago 15m 56.4M 2992M 19.0.0-2053-gb277feb2 668895944f0e fcd2e39a5866 2024-04-12T22:08:49.754 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5 smithi114 running (2m) 23s ago 14m 59.5M 2992M 19.0.0-2053-gb277feb2 668895944f0e 470c0780e428 2024-04-12T22:08:49.754 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6 smithi114 running (117s) 23s ago 14m 58.5M 2992M 19.0.0-2053-gb277feb2 668895944f0e 6b80aaeca08b 2024-04-12T22:08:49.754 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7 smithi114 running (83s) 23s ago 14m 55.5M 2992M 19.0.0-2053-gb277feb2 668895944f0e 1c3e102eaf74 2024-04-12T22:08:49.754 INFO:teuthology.orchestra.run.smithi097.stdout:prometheus.a smithi114 *:9095 running (6m) 23s ago 13m 72.0M - 2.33.4 514e6a882f6e 0ad5437ed63f 2024-04-12T22:08:49.754 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.foo.smithi097.sggtkh smithi097 *:8000 running (67s) 22s ago 11m 77.0M - 19.0.0-2053-gb277feb2 668895944f0e 96fa214b3dd8 2024-04-12T22:08:49.754 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.foo.smithi114.ulfbyo smithi114 *:8000 running (56s) 23s ago 11m 75.4M - 19.0.0-2053-gb277feb2 668895944f0e 02e33119b3ca 2024-04-12T22:08:49.754 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.smpl.smithi097.exxkgl smithi097 *:80 running (62s) 22s ago 11m 77.6M - 19.0.0-2053-gb277feb2 668895944f0e 7d210f6a20c9 2024-04-12T22:08:49.754 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.smpl.smithi114.lrgvph smithi114 *:80 running (50s) 23s ago 11m 75.0M - 19.0.0-2053-gb277feb2 668895944f0e 646ff8b6fed5 2024-04-12T22:08:50.388 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -e sha1=b277feb2c7beb4597d8ecc39d58d5c36b81f613a -- bash -c 'ceph versions' 2024-04-12T22:08:51.052 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:50 smithi097 ceph-mon[64853]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-12T22:08:51.052 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:50 smithi097 ceph-mon[64853]: from='client.44695 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:08:51.052 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:50 smithi097 ceph-mon[67407]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-12T22:08:51.052 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:50 smithi097 ceph-mon[67407]: from='client.44695 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:08:51.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:50 smithi114 ceph-mon[58780]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-12T22:08:51.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:50 smithi114 ceph-mon[58780]: from='client.44695 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:08:52.422 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-12T22:08:52.422 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": { 2024-04-12T22:08:52.422 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 3 2024-04-12T22:08:52.422 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:08:52.422 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": { 2024-04-12T22:08:52.422 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 2 2024-04-12T22:08:52.422 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:08:52.422 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": { 2024-04-12T22:08:52.422 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 8 2024-04-12T22:08:52.422 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:08:52.422 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw": { 2024-04-12T22:08:52.423 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 4 2024-04-12T22:08:52.423 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:08:52.423 INFO:teuthology.orchestra.run.smithi097.stdout: "overall": { 2024-04-12T22:08:52.423 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 17 2024-04-12T22:08:52.423 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-12T22:08:52.423 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-12T22:08:53.063 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:52 smithi097 ceph-mon[64853]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-04-12T22:08:53.063 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:52 smithi097 ceph-mon[64853]: from='client.? 172.21.15.97:0/2541385188' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:08:53.063 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:52 smithi097 ceph-mon[67407]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-04-12T22:08:53.063 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:52 smithi097 ceph-mon[67407]: from='client.? 172.21.15.97:0/2541385188' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:08:53.087 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -e sha1=b277feb2c7beb4597d8ecc39d58d5c36b81f613a -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-04-12T22:08:53.242 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:52 smithi114 ceph-mon[58780]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-04-12T22:08:53.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:52 smithi114 ceph-mon[58780]: from='client.? 172.21.15.97:0/2541385188' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:08:54.849 INFO:teuthology.orchestra.run.smithi097.stdout:wait for servicemap items w/ changing names to refresh 2024-04-12T22:08:54.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:54 smithi097 ceph-mon[64853]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:08:54.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:54 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:08:54.910 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:54 smithi097 ceph-mon[67407]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:08:54.910 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:54 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:08:55.242 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:54 smithi114 ceph-mon[58780]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:08:55.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:54 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:08:55.498 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -e sha1=b277feb2c7beb4597d8ecc39d58d5c36b81f613a -- bash -c 'sleep 60' 2024-04-12T22:08:56.090 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:55 smithi097 ceph-mon[64853]: from='client.54473 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:08:56.090 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:55 smithi097 ceph-mon[67407]: from='client.54473 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:08:56.144 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:55 smithi114 ceph-mon[58780]: from='client.54473 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:08:57.136 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:56 smithi097 ceph-mon[64853]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:08:57.136 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:56 smithi097 ceph-mon[67407]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:08:57.242 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:56 smithi114 ceph-mon[58780]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:08:59.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:58 smithi097 ceph-mon[64853]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:08:59.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:58 smithi097 ceph-mon[67407]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:08:59.242 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:58 smithi114 ceph-mon[58780]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:08:59.660 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:08:59 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:08:59] "GET /metrics HTTP/1.1" 200 38772 "" "Prometheus/2.33.4" 2024-04-12T22:09:00.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:08:59 smithi097 ceph-mon[67407]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:09:00.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:08:59 smithi097 ceph-mon[64853]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:09:00.242 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:08:59 smithi114 ceph-mon[58780]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:09:03.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:09:02 smithi097 ceph-mon[64853]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:09:03.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:09:02 smithi097 ceph-mon[67407]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:09:03.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:09:02 smithi114 ceph-mon[58780]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:09:05.162 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:09:04 smithi097 ceph-mon[64853]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:09:05.162 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:09:04 smithi097 ceph-mon[67407]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:09:05.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:09:04 smithi114 ceph-mon[58780]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:09:06.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:09:05 smithi097 ceph-mon[64853]: from='client.54473 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:09:06.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:09:05 smithi097 ceph-mon[67407]: from='client.54473 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:09:06.242 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:09:05 smithi114 ceph-mon[58780]: from='client.54473 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:09:07.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:09:06 smithi097 ceph-mon[64853]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:09:07.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:09:06 smithi097 ceph-mon[67407]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:09:07.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:09:06 smithi114 ceph-mon[58780]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:09:09.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:09:08 smithi097 ceph-mon[64853]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:09:09.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:09:08 smithi097 ceph-mon[67407]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:09:09.242 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:09:08 smithi114 ceph-mon[58780]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:09:09.662 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:09:09 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:09:09] "GET /metrics HTTP/1.1" 200 38772 "" "Prometheus/2.33.4" 2024-04-12T22:09:10.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:09:09 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:09:10.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:09:09 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:09:10.242 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:09:09 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:09:11.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:09:10 smithi097 ceph-mon[64853]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:09:11.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:09:10 smithi097 ceph-mon[67407]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:09:11.242 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:09:10 smithi114 ceph-mon[58780]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:09:13.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:09:12 smithi097 ceph-mon[64853]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:09:13.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:09:12 smithi097 ceph-mon[67407]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:09:13.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:09:12 smithi114 ceph-mon[58780]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:09:15.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:09:14 smithi097 ceph-mon[64853]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:09:15.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:09:14 smithi097 ceph-mon[67407]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:09:15.242 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:09:14 smithi114 ceph-mon[58780]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:09:16.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:09:15 smithi097 ceph-mon[64853]: from='client.54473 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:09:16.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:09:15 smithi097 ceph-mon[67407]: from='client.54473 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:09:16.242 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:09:15 smithi114 ceph-mon[58780]: from='client.54473 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:09:17.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:09:16 smithi097 ceph-mon[67407]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:09:17.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:09:16 smithi097 ceph-mon[64853]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:09:17.242 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:09:16 smithi114 ceph-mon[58780]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:09:19.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:09:18 smithi097 ceph-mon[67407]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:09:19.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:09:18 smithi097 ceph-mon[64853]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:09:19.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:09:18 smithi114 ceph-mon[58780]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:09:19.660 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:09:19 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:09:19] "GET /metrics HTTP/1.1" 200 38771 "" "Prometheus/2.33.4" 2024-04-12T22:09:21.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:09:20 smithi097 ceph-mon[64853]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:09:21.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:09:20 smithi097 ceph-mon[67407]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:09:21.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:09:20 smithi114 ceph-mon[58780]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:09:23.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:09:22 smithi097 ceph-mon[64853]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:09:23.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:09:22 smithi097 ceph-mon[67407]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:09:23.242 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:09:22 smithi114 ceph-mon[58780]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:09:25.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:09:24 smithi097 ceph-mon[67407]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:09:25.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:09:24 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:09:25.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:09:24 smithi097 ceph-mon[64853]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:09:25.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:09:24 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:09:25.242 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:09:24 smithi114 ceph-mon[58780]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:09:25.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:09:24 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:09:26.144 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:09:25 smithi114 ceph-mon[58780]: from='client.54473 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:09:26.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:09:25 smithi097 ceph-mon[64853]: from='client.54473 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:09:26.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:09:25 smithi097 ceph-mon[67407]: from='client.54473 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:09:27.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:09:26 smithi097 ceph-mon[64853]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:09:27.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:09:26 smithi097 ceph-mon[67407]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:09:27.242 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:09:26 smithi114 ceph-mon[58780]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:09:29.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:09:28 smithi097 ceph-mon[64853]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:09:29.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:09:28 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:09:29.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:09:28 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:09:29.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:09:28 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:09:29.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:09:28 smithi097 ceph-mon[67407]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:09:29.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:09:28 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:09:29.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:09:28 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:09:29.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:09:28 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:09:29.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:09:28 smithi114 ceph-mon[58780]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:09:29.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:09:28 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:09:29.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:09:28 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:09:29.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:09:28 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' 2024-04-12T22:09:29.660 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:09:29 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:09:29] "GET /metrics HTTP/1.1" 200 38772 "" "Prometheus/2.33.4" 2024-04-12T22:09:30.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:09:29 smithi097 ceph-mon[67407]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 958 B/s rd, 0 op/s 2024-04-12T22:09:30.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:09:29 smithi097 ceph-mon[67407]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:09:30.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:09:29 smithi097 ceph-mon[64853]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 958 B/s rd, 0 op/s 2024-04-12T22:09:30.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:09:29 smithi097 ceph-mon[64853]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:09:30.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:09:29 smithi114 ceph-mon[58780]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 958 B/s rd, 0 op/s 2024-04-12T22:09:30.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:09:29 smithi114 ceph-mon[58780]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:09:32.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:09:31 smithi097 ceph-mon[64853]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:09:32.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:09:31 smithi097 ceph-mon[67407]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:09:32.242 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:09:31 smithi114 ceph-mon[58780]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:09:34.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:09:33 smithi097 ceph-mon[67407]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:09:34.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:09:33 smithi097 ceph-mon[64853]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:09:34.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:09:33 smithi114 ceph-mon[58780]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:09:36.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:09:35 smithi097 ceph-mon[67407]: from='client.54473 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:09:36.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:09:35 smithi097 ceph-mon[67407]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:09:36.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:09:35 smithi097 ceph-mon[64853]: from='client.54473 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:09:36.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:09:35 smithi097 ceph-mon[64853]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:09:36.242 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:09:35 smithi114 ceph-mon[58780]: from='client.54473 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:09:36.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:09:35 smithi114 ceph-mon[58780]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:09:38.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:09:37 smithi097 ceph-mon[64853]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 589 B/s rd, 0 op/s 2024-04-12T22:09:38.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:09:37 smithi097 ceph-mon[67407]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 589 B/s rd, 0 op/s 2024-04-12T22:09:38.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:09:37 smithi114 ceph-mon[58780]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 589 B/s rd, 0 op/s 2024-04-12T22:09:39.660 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:09:39 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:09:39] "GET /metrics HTTP/1.1" 200 38772 "" "Prometheus/2.33.4" 2024-04-12T22:09:40.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:09:39 smithi097 ceph-mon[67407]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-12T22:09:40.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:09:39 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:09:40.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:09:39 smithi097 ceph-mon[64853]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-12T22:09:40.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:09:39 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:09:40.242 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:09:39 smithi114 ceph-mon[58780]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-12T22:09:40.242 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:09:39 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:09:42.159 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:09:41 smithi097 ceph-mon[64853]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:09:42.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:09:41 smithi097 ceph-mon[67407]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:09:42.242 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:09:41 smithi114 ceph-mon[58780]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:09:44.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:09:43 smithi097 ceph-mon[64853]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:09:44.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:09:43 smithi097 ceph-mon[67407]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:09:44.242 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:09:43 smithi114 ceph-mon[58780]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:09:46.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:09:45 smithi097 ceph-mon[64853]: from='client.54473 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:09:46.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:09:45 smithi097 ceph-mon[64853]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:09:46.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:09:45 smithi097 ceph-mon[67407]: from='client.54473 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:09:46.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:09:45 smithi097 ceph-mon[67407]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:09:46.242 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:09:45 smithi114 ceph-mon[58780]: from='client.54473 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:09:46.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:09:45 smithi114 ceph-mon[58780]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:09:48.160 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:09:47 smithi097 ceph-mon[64853]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:09:48.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:09:47 smithi097 ceph-mon[67407]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:09:48.242 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:09:47 smithi114 ceph-mon[58780]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:09:49.193 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:09:48 smithi097 ceph-mon[67407]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:09:49.193 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:09:48 smithi097 ceph-mon[64853]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:09:49.242 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:09:48 smithi114 ceph-mon[58780]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:09:49.660 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:09:49 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:09:49] "GET /metrics HTTP/1.1" 200 38769 "" "Prometheus/2.33.4" 2024-04-12T22:09:51.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:09:51 smithi097 ceph-mon[64853]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:09:51.910 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:09:51 smithi097 ceph-mon[67407]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:09:51.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:09:51 smithi114 ceph-mon[58780]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:09:53.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:09:53 smithi097 ceph-mon[64853]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:09:53.910 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:09:53 smithi097 ceph-mon[67407]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:09:53.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:09:53 smithi114 ceph-mon[58780]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:09:55.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:09:55 smithi097 ceph-mon[64853]: from='client.54473 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:09:55.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:09:55 smithi097 ceph-mon[64853]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:09:55.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:09:55 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:09:55.910 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:09:55 smithi097 ceph-mon[67407]: from='client.54473 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:09:55.910 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:09:55 smithi097 ceph-mon[67407]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:09:55.910 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:09:55 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:09:55.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:09:55 smithi114 ceph-mon[58780]: from='client.54473 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:09:55.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:09:55 smithi114 ceph-mon[58780]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:09:55.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:09:55 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:09:57.627 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:09:57 smithi097 ceph-mon[64853]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:09:57.627 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:09:57 smithi097 ceph-mon[67407]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:09:57.794 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:09:57 smithi114 ceph-mon[58780]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:09:58.064 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -e sha1=b277feb2c7beb4597d8ecc39d58d5c36b81f613a -- bash -c 'ceph orch ps' 2024-04-12T22:09:59.486 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:09:59 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:09:59] "GET /metrics HTTP/1.1" 200 38771 "" "Prometheus/2.33.4" 2024-04-12T22:09:59.760 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:09:59 smithi097 ceph-mon[64853]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:09:59.761 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:09:59 smithi097 ceph-mon[67407]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:09:59.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:09:59 smithi114 ceph-mon[58780]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:10:00.226 INFO:teuthology.orchestra.run.smithi097.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T22:10:00.227 INFO:teuthology.orchestra.run.smithi097.stdout:alertmanager.a smithi097 *:9093,9094 running (8m) 92s ago 14m 24.1M - 0.23.0 ba2b418f427c 7b4c491fe0a0 2024-04-12T22:10:00.227 INFO:teuthology.orchestra.run.smithi097.stdout:grafana.a smithi114 *:3000 error 93s ago 13m - - 2024-04-12T22:10:00.227 INFO:teuthology.orchestra.run.smithi097.stdout:iscsi.foo.smithi097.xndkxy smithi097 unknown 92s ago 12m - - 2024-04-12T22:10:00.227 INFO:teuthology.orchestra.run.smithi097.stdout:mgr.x smithi114 *:8443,9283,8765 running (7m) 93s ago 18m 452M - 19.0.0-2053-gb277feb2 668895944f0e b8f8df3bf928 2024-04-12T22:10:00.227 INFO:teuthology.orchestra.run.smithi097.stdout:mgr.y smithi097 *:8443,9283,8765 running (8m) 92s ago 20m 514M - 19.0.0-2053-gb277feb2 668895944f0e 488170dac11f 2024-04-12T22:10:00.227 INFO:teuthology.orchestra.run.smithi097.stdout:mon.a smithi097 running (7m) 92s ago 20m 60.9M 2048M 19.0.0-2053-gb277feb2 668895944f0e 3cfa16d3a85c 2024-04-12T22:10:00.227 INFO:teuthology.orchestra.run.smithi097.stdout:mon.b smithi114 running (6m) 93s ago 18m 44.5M 2048M 19.0.0-2053-gb277feb2 668895944f0e 71a2eff159a3 2024-04-12T22:10:00.227 INFO:teuthology.orchestra.run.smithi097.stdout:mon.c smithi097 running (7m) 92s ago 18m 48.0M 2048M 19.0.0-2053-gb277feb2 668895944f0e 69507ba88603 2024-04-12T22:10:00.227 INFO:teuthology.orchestra.run.smithi097.stdout:node-exporter.a smithi097 *:9100 running (9m) 92s ago 14m 18.4M - 1.3.1 1dbe0e931976 5edff9053b37 2024-04-12T22:10:00.227 INFO:teuthology.orchestra.run.smithi097.stdout:node-exporter.b smithi114 *:9100 running (8m) 93s ago 14m 17.1M - 1.3.1 1dbe0e931976 904afc45dc2f 2024-04-12T22:10:00.227 INFO:teuthology.orchestra.run.smithi097.stdout:osd.0 smithi097 running (5m) 92s ago 18m 64.0M 2736M 19.0.0-2053-gb277feb2 668895944f0e 9d05fea4bf37 2024-04-12T22:10:00.227 INFO:teuthology.orchestra.run.smithi097.stdout:osd.1 smithi097 running (5m) 92s ago 17m 56.5M 2736M 19.0.0-2053-gb277feb2 668895944f0e a9a527642889 2024-04-12T22:10:00.228 INFO:teuthology.orchestra.run.smithi097.stdout:osd.2 smithi097 running (4m) 92s ago 17m 52.7M 2736M 19.0.0-2053-gb277feb2 668895944f0e 2e2297afff50 2024-04-12T22:10:00.228 INFO:teuthology.orchestra.run.smithi097.stdout:osd.3 smithi097 running (4m) 92s ago 16m 62.3M 2736M 19.0.0-2053-gb277feb2 668895944f0e 415b4d3d85f2 2024-04-12T22:10:00.228 INFO:teuthology.orchestra.run.smithi097.stdout:osd.4 smithi114 running (4m) 93s ago 16m 56.4M 2992M 19.0.0-2053-gb277feb2 668895944f0e fcd2e39a5866 2024-04-12T22:10:00.228 INFO:teuthology.orchestra.run.smithi097.stdout:osd.5 smithi114 running (3m) 93s ago 16m 59.5M 2992M 19.0.0-2053-gb277feb2 668895944f0e 470c0780e428 2024-04-12T22:10:00.228 INFO:teuthology.orchestra.run.smithi097.stdout:osd.6 smithi114 running (3m) 93s ago 15m 58.5M 2992M 19.0.0-2053-gb277feb2 668895944f0e 6b80aaeca08b 2024-04-12T22:10:00.228 INFO:teuthology.orchestra.run.smithi097.stdout:osd.7 smithi114 running (2m) 93s ago 15m 55.5M 2992M 19.0.0-2053-gb277feb2 668895944f0e 1c3e102eaf74 2024-04-12T22:10:00.228 INFO:teuthology.orchestra.run.smithi097.stdout:prometheus.a smithi114 *:9095 running (7m) 93s ago 14m 72.0M - 2.33.4 514e6a882f6e 0ad5437ed63f 2024-04-12T22:10:00.228 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.foo.smithi097.sggtkh smithi097 *:8000 running (2m) 92s ago 13m 77.0M - 19.0.0-2053-gb277feb2 668895944f0e 96fa214b3dd8 2024-04-12T22:10:00.228 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.foo.smithi114.ulfbyo smithi114 *:8000 running (2m) 93s ago 13m 75.4M - 19.0.0-2053-gb277feb2 668895944f0e 02e33119b3ca 2024-04-12T22:10:00.228 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.smpl.smithi097.exxkgl smithi097 *:80 running (2m) 92s ago 12m 77.6M - 19.0.0-2053-gb277feb2 668895944f0e 7d210f6a20c9 2024-04-12T22:10:00.228 INFO:teuthology.orchestra.run.smithi097.stdout:rgw.smpl.smithi114.lrgvph smithi114 *:80 running (2m) 93s ago 12m 75.0M - 19.0.0-2053-gb277feb2 668895944f0e 646ff8b6fed5 2024-04-12T22:10:00.886 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:00 smithi097 ceph-mon[64853]: Health detail: HEALTH_WARN 2 failed cephadm daemon(s); Upgrading daemon iscsi.foo.smithi097.xndkxy on host smithi097 failed. 2024-04-12T22:10:00.887 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:00 smithi097 ceph-mon[64853]: [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-04-12T22:10:00.887 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:00 smithi097 ceph-mon[64853]: daemon iscsi.foo.smithi097.xndkxy on smithi097 is in unknown state 2024-04-12T22:10:00.887 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:00 smithi097 ceph-mon[64853]: daemon grafana.a on smithi114 is in error state 2024-04-12T22:10:00.887 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:00 smithi097 ceph-mon[64853]: [WRN] UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi097.xndkxy on host smithi097 failed. 2024-04-12T22:10:00.887 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:00 smithi097 ceph-mon[64853]: Upgrade daemon: iscsi.foo.smithi097.xndkxy: cephadm exited with an error code: 1, stderr: Redeploy daemon iscsi.foo.smithi097.xndkxy ... 2024-04-12T22:10:00.887 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:00 smithi097 ceph-mon[64853]: Creating ceph-iscsi config... 2024-04-12T22:10:00.887 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:00 smithi097 ceph-mon[64853]: Write file: /var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/iscsi.foo.smithi097.xndkxy/iscsi-gateway.cfg 2024-04-12T22:10:00.887 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:00 smithi097 ceph-mon[64853]: Write file: /var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/iscsi.foo.smithi097.xndkxy/tcmu-runner-entrypoint.sh 2024-04-12T22:10:00.887 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:00 smithi097 ceph-mon[64853]: Failed to trim old cgroups /sys/fs/cgroup/system.slice/system-ceph\x2d80bf7680\x2df916\x2d11ee\x2dbc8d\x2dc7b262605968.slice/ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service 2024-04-12T22:10:00.887 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:00 smithi097 ceph-mon[64853]: Non-zero exit code 1 from systemctl start ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy 2024-04-12T22:10:00.887 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:00 smithi097 ceph-mon[64853]: systemctl: stderr Job for ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service failed because the control process exited with error code. 2024-04-12T22:10:00.888 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:00 smithi097 ceph-mon[64853]: systemctl: stderr See "systemctl status ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service" and "journalctl -xeu ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service" for details. 2024-04-12T22:10:00.888 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:00 smithi097 ceph-mon[64853]: Traceback (most recent call last): 2024-04-12T22:10:00.888 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:00 smithi097 ceph-mon[64853]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-12T22:10:00.888 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:00 smithi097 ceph-mon[64853]: return _run_code(code, main_globals, None, 2024-04-12T22:10:00.888 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:00 smithi097 ceph-mon[64853]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-12T22:10:00.888 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:00 smithi097 ceph-mon[64853]: exec(code, run_globals) 2024-04-12T22:10:00.888 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:00 smithi097 ceph-mon[64853]: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 5534, in 2024-04-12T22:10:00.888 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:00 smithi097 ceph-mon[64853]: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 5522, in main 2024-04-12T22:10:00.888 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:00 smithi097 ceph-mon[64853]: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 3045, in command_deploy_from 2024-04-12T22:10:00.888 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:00 smithi097 ceph-mon[64853]: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 3080, in _common_deploy 2024-04-12T22:10:00.888 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:00 smithi097 ceph-mon[64853]: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 3100, in _deploy_daemon_container 2024-04-12T22:10:00.888 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:00 smithi097 ceph-mon[64853]: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 1036, in deploy_daemon 2024-04-12T22:10:00.888 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:00 smithi097 ceph-mon[64853]: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 1172, in deploy_daemon_units 2024-04-12T22:10:00.889 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:00 smithi097 ceph-mon[67407]: Health detail: HEALTH_WARN 2 failed cephadm daemon(s); Upgrading daemon iscsi.foo.smithi097.xndkxy on host smithi097 failed. 2024-04-12T22:10:00.889 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:00 smithi097 ceph-mon[67407]: [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-04-12T22:10:00.889 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:00 smithi097 ceph-mon[67407]: daemon iscsi.foo.smithi097.xndkxy on smithi097 is in unknown state 2024-04-12T22:10:00.889 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:00 smithi097 ceph-mon[67407]: daemon grafana.a on smithi114 is in error state 2024-04-12T22:10:00.889 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:00 smithi097 ceph-mon[67407]: [WRN] UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi097.xndkxy on host smithi097 failed. 2024-04-12T22:10:00.889 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:00 smithi097 ceph-mon[67407]: Upgrade daemon: iscsi.foo.smithi097.xndkxy: cephadm exited with an error code: 1, stderr: Redeploy daemon iscsi.foo.smithi097.xndkxy ... 2024-04-12T22:10:00.889 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:00 smithi097 ceph-mon[67407]: Creating ceph-iscsi config... 2024-04-12T22:10:00.889 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:00 smithi097 ceph-mon[67407]: Write file: /var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/iscsi.foo.smithi097.xndkxy/iscsi-gateway.cfg 2024-04-12T22:10:00.889 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:00 smithi097 ceph-mon[67407]: Write file: /var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/iscsi.foo.smithi097.xndkxy/tcmu-runner-entrypoint.sh 2024-04-12T22:10:00.889 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:00 smithi097 ceph-mon[67407]: Failed to trim old cgroups /sys/fs/cgroup/system.slice/system-ceph\x2d80bf7680\x2df916\x2d11ee\x2dbc8d\x2dc7b262605968.slice/ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service 2024-04-12T22:10:00.889 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:00 smithi097 ceph-mon[67407]: Non-zero exit code 1 from systemctl start ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy 2024-04-12T22:10:00.889 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:00 smithi097 ceph-mon[67407]: systemctl: stderr Job for ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service failed because the control process exited with error code. 2024-04-12T22:10:00.890 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:00 smithi097 ceph-mon[67407]: systemctl: stderr See "systemctl status ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service" and "journalctl -xeu ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service" for details. 2024-04-12T22:10:00.890 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:00 smithi097 ceph-mon[67407]: Traceback (most recent call last): 2024-04-12T22:10:00.890 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:00 smithi097 ceph-mon[67407]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-12T22:10:00.890 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:00 smithi097 ceph-mon[67407]: return _run_code(code, main_globals, None, 2024-04-12T22:10:00.890 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:00 smithi097 ceph-mon[67407]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-12T22:10:00.890 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:00 smithi097 ceph-mon[67407]: exec(code, run_globals) 2024-04-12T22:10:00.890 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:00 smithi097 ceph-mon[67407]: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 5534, in 2024-04-12T22:10:00.890 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:00 smithi097 ceph-mon[67407]: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 5522, in main 2024-04-12T22:10:00.890 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:00 smithi097 ceph-mon[67407]: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 3045, in command_deploy_from 2024-04-12T22:10:00.890 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:00 smithi097 ceph-mon[67407]: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 3080, in _common_deploy 2024-04-12T22:10:00.890 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:00 smithi097 ceph-mon[67407]: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 3100, in _deploy_daemon_container 2024-04-12T22:10:00.890 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:00 smithi097 ceph-mon[67407]: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 1036, in deploy_daemon 2024-04-12T22:10:00.890 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:00 smithi097 ceph-mon[67407]: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 1172, in deploy_daemon_units 2024-04-12T22:10:00.891 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:00 smithi097 ceph-mon[64853]: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-12T22:10:00.891 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:00 smithi097 ceph-mon[64853]: RuntimeError: Failed command: systemctl start ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy: Job for ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service failed because the control process exited with error code. 2024-04-12T22:10:00.891 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:00 smithi097 ceph-mon[64853]: See "systemctl status ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service" and "journalctl -xeu ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service" for details. 2024-04-12T22:10:00.891 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:00 smithi097 ceph-mon[67407]: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-12T22:10:00.891 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:00 smithi097 ceph-mon[67407]: RuntimeError: Failed command: systemctl start ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy: Job for ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service failed because the control process exited with error code. 2024-04-12T22:10:00.891 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:00 smithi097 ceph-mon[67407]: See "systemctl status ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service" and "journalctl -xeu ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service" for details. 2024-04-12T22:10:00.910 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -e sha1=b277feb2c7beb4597d8ecc39d58d5c36b81f613a -- bash -c 'ceph versions' 2024-04-12T22:10:00.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:00 smithi114 ceph-mon[58780]: Health detail: HEALTH_WARN 2 failed cephadm daemon(s); Upgrading daemon iscsi.foo.smithi097.xndkxy on host smithi097 failed. 2024-04-12T22:10:00.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:00 smithi114 ceph-mon[58780]: [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-04-12T22:10:00.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:00 smithi114 ceph-mon[58780]: daemon iscsi.foo.smithi097.xndkxy on smithi097 is in unknown state 2024-04-12T22:10:00.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:00 smithi114 ceph-mon[58780]: daemon grafana.a on smithi114 is in error state 2024-04-12T22:10:00.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:00 smithi114 ceph-mon[58780]: [WRN] UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi097.xndkxy on host smithi097 failed. 2024-04-12T22:10:00.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:00 smithi114 ceph-mon[58780]: Upgrade daemon: iscsi.foo.smithi097.xndkxy: cephadm exited with an error code: 1, stderr: Redeploy daemon iscsi.foo.smithi097.xndkxy ... 2024-04-12T22:10:00.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:00 smithi114 ceph-mon[58780]: Creating ceph-iscsi config... 2024-04-12T22:10:00.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:00 smithi114 ceph-mon[58780]: Write file: /var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/iscsi.foo.smithi097.xndkxy/iscsi-gateway.cfg 2024-04-12T22:10:00.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:00 smithi114 ceph-mon[58780]: Write file: /var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/iscsi.foo.smithi097.xndkxy/tcmu-runner-entrypoint.sh 2024-04-12T22:10:00.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:00 smithi114 ceph-mon[58780]: Failed to trim old cgroups /sys/fs/cgroup/system.slice/system-ceph\x2d80bf7680\x2df916\x2d11ee\x2dbc8d\x2dc7b262605968.slice/ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service 2024-04-12T22:10:00.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:00 smithi114 ceph-mon[58780]: Non-zero exit code 1 from systemctl start ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy 2024-04-12T22:10:00.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:00 smithi114 ceph-mon[58780]: systemctl: stderr Job for ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service failed because the control process exited with error code. 2024-04-12T22:10:00.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:00 smithi114 ceph-mon[58780]: systemctl: stderr See "systemctl status ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service" and "journalctl -xeu ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service" for details. 2024-04-12T22:10:00.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:00 smithi114 ceph-mon[58780]: Traceback (most recent call last): 2024-04-12T22:10:00.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:00 smithi114 ceph-mon[58780]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-12T22:10:00.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:00 smithi114 ceph-mon[58780]: return _run_code(code, main_globals, None, 2024-04-12T22:10:00.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:00 smithi114 ceph-mon[58780]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-12T22:10:00.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:00 smithi114 ceph-mon[58780]: exec(code, run_globals) 2024-04-12T22:10:00.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:00 smithi114 ceph-mon[58780]: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 5534, in 2024-04-12T22:10:00.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:00 smithi114 ceph-mon[58780]: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 5522, in main 2024-04-12T22:10:00.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:00 smithi114 ceph-mon[58780]: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 3045, in command_deploy_from 2024-04-12T22:10:00.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:00 smithi114 ceph-mon[58780]: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 3080, in _common_deploy 2024-04-12T22:10:00.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:00 smithi114 ceph-mon[58780]: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 3100, in _deploy_daemon_container 2024-04-12T22:10:00.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:00 smithi114 ceph-mon[58780]: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 1036, in deploy_daemon 2024-04-12T22:10:00.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:00 smithi114 ceph-mon[58780]: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 1172, in deploy_daemon_units 2024-04-12T22:10:00.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:00 smithi114 ceph-mon[58780]: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-12T22:10:00.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:00 smithi114 ceph-mon[58780]: RuntimeError: Failed command: systemctl start ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy: Job for ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service failed because the control process exited with error code. 2024-04-12T22:10:00.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:00 smithi114 ceph-mon[58780]: See "systemctl status ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service" and "journalctl -xeu ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service" for details. 2024-04-12T22:10:01.910 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:01 smithi097 ceph-mon[67407]: from='client.34776 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:10:01.910 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:01 smithi097 ceph-mon[67407]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:10:01.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:01 smithi097 ceph-mon[64853]: from='client.34776 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:10:01.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:01 smithi097 ceph-mon[64853]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:10:01.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:01 smithi114 ceph-mon[58780]: from='client.34776 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:10:01.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:01 smithi114 ceph-mon[58780]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:10:03.127 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-12T22:10:03.128 INFO:teuthology.orchestra.run.smithi097.stdout: "mon": { 2024-04-12T22:10:03.128 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 3 2024-04-12T22:10:03.128 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:10:03.128 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr": { 2024-04-12T22:10:03.128 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 2 2024-04-12T22:10:03.128 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:10:03.128 INFO:teuthology.orchestra.run.smithi097.stdout: "osd": { 2024-04-12T22:10:03.128 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 8 2024-04-12T22:10:03.128 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:10:03.128 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw": { 2024-04-12T22:10:03.128 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 4 2024-04-12T22:10:03.128 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-04-12T22:10:03.128 INFO:teuthology.orchestra.run.smithi097.stdout: "overall": { 2024-04-12T22:10:03.128 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)": 17 2024-04-12T22:10:03.128 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-04-12T22:10:03.129 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-12T22:10:03.773 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:03 smithi097 ceph-mon[64853]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:10:03.773 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:03 smithi097 ceph-mon[64853]: from='client.? 172.21.15.97:0/1918545052' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:10:03.773 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:03 smithi097 ceph-mon[67407]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:10:03.773 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:03 smithi097 ceph-mon[67407]: from='client.? 172.21.15.97:0/1918545052' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:10:03.775 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -e sha1=b277feb2c7beb4597d8ecc39d58d5c36b81f613a -- bash -c 'ceph orch upgrade status' 2024-04-12T22:10:03.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:03 smithi114 ceph-mon[58780]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:10:03.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:03 smithi114 ceph-mon[58780]: from='client.? 172.21.15.97:0/1918545052' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:10:05.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:05 smithi097 ceph-mon[64853]: from='client.54473 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:10:05.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:05 smithi097 ceph-mon[64853]: pgmap v302: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:10:05.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:05 smithi097 ceph-mon[67407]: from='client.54473 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:10:05.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:05 smithi097 ceph-mon[67407]: pgmap v302: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:10:05.806 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-12T22:10:05.806 INFO:teuthology.orchestra.run.smithi097.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a", 2024-04-12T22:10:05.806 INFO:teuthology.orchestra.run.smithi097.stdout: "in_progress": true, 2024-04-12T22:10:05.806 INFO:teuthology.orchestra.run.smithi097.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-12T22:10:05.806 INFO:teuthology.orchestra.run.smithi097.stdout: "services_complete": [ 2024-04-12T22:10:05.806 INFO:teuthology.orchestra.run.smithi097.stdout: "rgw", 2024-04-12T22:10:05.806 INFO:teuthology.orchestra.run.smithi097.stdout: "mon", 2024-04-12T22:10:05.806 INFO:teuthology.orchestra.run.smithi097.stdout: "mgr", 2024-04-12T22:10:05.806 INFO:teuthology.orchestra.run.smithi097.stdout: "osd" 2024-04-12T22:10:05.806 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-04-12T22:10:05.806 INFO:teuthology.orchestra.run.smithi097.stdout: "progress": "17/23 daemons upgraded", 2024-04-12T22:10:05.806 INFO:teuthology.orchestra.run.smithi097.stdout: "message": "Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi097.xndkxy on host smithi097 failed.", 2024-04-12T22:10:05.807 INFO:teuthology.orchestra.run.smithi097.stdout: "is_paused": true 2024-04-12T22:10:05.807 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-12T22:10:05.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:05 smithi114 ceph-mon[58780]: from='client.54473 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:10:05.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:05 smithi114 ceph-mon[58780]: pgmap v302: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:10:06.500 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -e sha1=b277feb2c7beb4597d8ecc39d58d5c36b81f613a -- bash -c 'ceph health detail' 2024-04-12T22:10:06.807 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:06 smithi097 ceph-mon[64853]: from='client.34788 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:10:06.807 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:06 smithi097 ceph-mon[67407]: from='client.34788 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:10:06.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:06 smithi114 ceph-mon[58780]: from='client.34788 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:10:07.792 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:07 smithi097 ceph-mon[64853]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:10:07.792 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:07 smithi097 ceph-mon[67407]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:10:07.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:07 smithi114 ceph-mon[58780]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:10:08.554 INFO:teuthology.orchestra.run.smithi097.stdout:HEALTH_WARN 2 failed cephadm daemon(s); Upgrading daemon iscsi.foo.smithi097.xndkxy on host smithi097 failed. 2024-04-12T22:10:08.554 INFO:teuthology.orchestra.run.smithi097.stdout:[WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-04-12T22:10:08.554 INFO:teuthology.orchestra.run.smithi097.stdout: daemon iscsi.foo.smithi097.xndkxy on smithi097 is in unknown state 2024-04-12T22:10:08.555 INFO:teuthology.orchestra.run.smithi097.stdout: daemon grafana.a on smithi114 is in error state 2024-04-12T22:10:08.555 INFO:teuthology.orchestra.run.smithi097.stdout:[WRN] UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi097.xndkxy on host smithi097 failed. 2024-04-12T22:10:08.555 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrade daemon: iscsi.foo.smithi097.xndkxy: cephadm exited with an error code: 1, stderr: Redeploy daemon iscsi.foo.smithi097.xndkxy ... 2024-04-12T22:10:08.555 INFO:teuthology.orchestra.run.smithi097.stdout:Creating ceph-iscsi config... 2024-04-12T22:10:08.555 INFO:teuthology.orchestra.run.smithi097.stdout:Write file: /var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/iscsi.foo.smithi097.xndkxy/iscsi-gateway.cfg 2024-04-12T22:10:08.555 INFO:teuthology.orchestra.run.smithi097.stdout:Write file: /var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/iscsi.foo.smithi097.xndkxy/tcmu-runner-entrypoint.sh 2024-04-12T22:10:08.555 INFO:teuthology.orchestra.run.smithi097.stdout:Failed to trim old cgroups /sys/fs/cgroup/system.slice/system-ceph\x2d80bf7680\x2df916\x2d11ee\x2dbc8d\x2dc7b262605968.slice/ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service 2024-04-12T22:10:08.555 INFO:teuthology.orchestra.run.smithi097.stdout:Non-zero exit code 1 from systemctl start ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy 2024-04-12T22:10:08.555 INFO:teuthology.orchestra.run.smithi097.stdout:systemctl: stderr Job for ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service failed because the control process exited with error code. 2024-04-12T22:10:08.555 INFO:teuthology.orchestra.run.smithi097.stdout:systemctl: stderr See "systemctl status ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service" and "journalctl -xeu ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service" for details. 2024-04-12T22:10:08.555 INFO:teuthology.orchestra.run.smithi097.stdout:Traceback (most recent call last): 2024-04-12T22:10:08.555 INFO:teuthology.orchestra.run.smithi097.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-12T22:10:08.555 INFO:teuthology.orchestra.run.smithi097.stdout: return _run_code(code, main_globals, None, 2024-04-12T22:10:08.555 INFO:teuthology.orchestra.run.smithi097.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-12T22:10:08.555 INFO:teuthology.orchestra.run.smithi097.stdout: exec(code, run_globals) 2024-04-12T22:10:08.556 INFO:teuthology.orchestra.run.smithi097.stdout: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 5534, in 2024-04-12T22:10:08.556 INFO:teuthology.orchestra.run.smithi097.stdout: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 5522, in main 2024-04-12T22:10:08.556 INFO:teuthology.orchestra.run.smithi097.stdout: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 3045, in command_deploy_from 2024-04-12T22:10:08.556 INFO:teuthology.orchestra.run.smithi097.stdout: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 3080, in _common_deploy 2024-04-12T22:10:08.556 INFO:teuthology.orchestra.run.smithi097.stdout: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 3100, in _deploy_daemon_container 2024-04-12T22:10:08.556 INFO:teuthology.orchestra.run.smithi097.stdout: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 1036, in deploy_daemon 2024-04-12T22:10:08.556 INFO:teuthology.orchestra.run.smithi097.stdout: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/__main__.py", line 1172, in deploy_daemon_units 2024-04-12T22:10:08.556 INFO:teuthology.orchestra.run.smithi097.stdout: File "/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/cephadm.3d7e653875625f0c77c19c6eec634e5e96c6029bf8db67dbe4f37726f33fe44e/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-12T22:10:08.556 INFO:teuthology.orchestra.run.smithi097.stdout:RuntimeError: Failed command: systemctl start ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy: Job for ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service failed because the control process exited with error code. 2024-04-12T22:10:08.556 INFO:teuthology.orchestra.run.smithi097.stdout:See "systemctl status ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service" and "journalctl -xeu ceph-80bf7680-f916-11ee-bc8d-c7b262605968@iscsi.foo.smithi097.xndkxy.service" for details. 2024-04-12T22:10:09.319 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -e sha1=b277feb2c7beb4597d8ecc39d58d5c36b81f613a -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-04-12T22:10:09.529 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:10:09 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:10:09] "GET /metrics HTTP/1.1" 200 38771 "" "Prometheus/2.33.4" 2024-04-12T22:10:09.848 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:09 smithi097 ceph-mon[64853]: pgmap v304: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:10:09.848 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:09 smithi097 ceph-mon[64853]: from='client.? 172.21.15.97:0/4199188858' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T22:10:09.848 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:09 smithi097 ceph-mon[67407]: pgmap v304: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:10:09.848 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:09 smithi097 ceph-mon[67407]: from='client.? 172.21.15.97:0/4199188858' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T22:10:09.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:09 smithi114 ceph-mon[58780]: pgmap v304: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:10:09.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:09 smithi114 ceph-mon[58780]: from='client.? 172.21.15.97:0/4199188858' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-12T22:10:10.601 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:10 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:10:10.601 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:10 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:10:10.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:10 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:10:11.395 INFO:teuthology.orchestra.run.smithi097.stdout:true 2024-04-12T22:10:11.876 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:11 smithi097 ceph-mon[64853]: pgmap v305: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:10:11.876 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:11 smithi097 ceph-mon[64853]: from='client.? 172.21.15.97:0/2379815889' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:10:11.876 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:11 smithi097 ceph-mon[67407]: pgmap v305: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:10:11.876 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:11 smithi097 ceph-mon[67407]: from='client.? 172.21.15.97:0/2379815889' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:10:11.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:11 smithi114 ceph-mon[58780]: pgmap v305: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:10:11.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:11 smithi114 ceph-mon[58780]: from='client.? 172.21.15.97:0/2379815889' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:10:12.110 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -e sha1=b277feb2c7beb4597d8ecc39d58d5c36b81f613a -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-04-12T22:10:13.824 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:13 smithi097 ceph-mon[64853]: pgmap v306: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:10:13.824 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:13 smithi097 ceph-mon[67407]: pgmap v306: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:10:13.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:13 smithi114 ceph-mon[58780]: pgmap v306: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:10:14.324 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev)" 2024-04-12T22:10:14.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:14 smithi097 ceph-mon[64853]: from='client.? 172.21.15.97:0/2327597027' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:10:14.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:14 smithi097 ceph-mon[67407]: from='client.? 172.21.15.97:0/2327597027' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:10:14.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:14 smithi114 ceph-mon[58780]: from='client.? 172.21.15.97:0/2327597027' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-12T22:10:15.034 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -e sha1=b277feb2c7beb4597d8ecc39d58d5c36b81f613a -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2024-04-12T22:10:15.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:15 smithi097 ceph-mon[64853]: from='client.54473 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:10:15.660 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:15 smithi097 ceph-mon[64853]: pgmap v307: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:10:15.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:15 smithi097 ceph-mon[67407]: from='client.54473 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:10:15.660 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:15 smithi097 ceph-mon[67407]: pgmap v307: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:10:15.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:15 smithi114 ceph-mon[58780]: from='client.54473 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:10:15.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:15 smithi114 ceph-mon[58780]: pgmap v307: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:10:17.148 INFO:teuthology.orchestra.run.smithi097.stdout:osd 8 110s ago - 2024-04-12T22:10:17.802 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:17 smithi097 ceph-mon[64853]: pgmap v308: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:10:17.802 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:17 smithi097 ceph-mon[67407]: pgmap v308: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:10:17.827 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-12T22:10:17.836 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi097.front.sepia.ceph.com 2024-04-12T22:10:17.837 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- bash -c 'ceph orch upgrade ls' 2024-04-12T22:10:17.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:17 smithi114 ceph-mon[58780]: pgmap v308: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:10:18.869 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:18 smithi097 ceph-mon[64853]: from='client.34812 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:10:18.870 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:18 smithi097 ceph-mon[67407]: from='client.34812 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:10:18.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:18 smithi114 ceph-mon[58780]: from='client.34812 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:10:19.550 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:10:19 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:10:19] "GET /metrics HTTP/1.1" 200 38774 "" "Prometheus/2.33.4" 2024-04-12T22:10:19.909 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:19 smithi097 ceph-mon[64853]: pgmap v309: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:10:19.910 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:19 smithi097 ceph-mon[67407]: pgmap v309: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:10:19.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:19 smithi114 ceph-mon[58780]: pgmap v309: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:10:20.625 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-04-12T22:10:20.625 INFO:teuthology.orchestra.run.smithi097.stdout: "image": "quay.io/ceph/ceph", 2024-04-12T22:10:20.625 INFO:teuthology.orchestra.run.smithi097.stdout: "registry": "quay.io", 2024-04-12T22:10:20.625 INFO:teuthology.orchestra.run.smithi097.stdout: "bare_image": "ceph/ceph", 2024-04-12T22:10:20.625 INFO:teuthology.orchestra.run.smithi097.stdout: "versions": [] 2024-04-12T22:10:20.625 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-04-12T22:10:21.339 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0' 2024-04-12T22:10:21.656 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:21 smithi097 ceph-mon[64853]: from='client.34818 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:10:21.656 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:21 smithi097 ceph-mon[64853]: pgmap v310: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:10:21.656 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:21 smithi097 ceph-mon[67407]: from='client.34818 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:10:21.656 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:21 smithi097 ceph-mon[67407]: pgmap v310: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:10:21.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:21 smithi114 ceph-mon[58780]: from='client.34818 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:10:21.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:21 smithi114 ceph-mon[58780]: pgmap v310: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:10:23.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:23 smithi097 ceph-mon[64853]: pgmap v311: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:10:23.910 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:23 smithi097 ceph-mon[67407]: pgmap v311: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:10:23.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:23 smithi114 ceph-mon[58780]: pgmap v311: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T22:10:24.150 INFO:teuthology.orchestra.run.smithi097.stdout: "16.2.0", 2024-04-12T22:10:24.752 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:24 smithi097 ceph-mon[64853]: from='client.44755 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:10:24.752 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:24 smithi097 ceph-mon[67407]: from='client.44755 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:10:24.776 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2' 2024-04-12T22:10:24.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:24 smithi114 ceph-mon[58780]: from='client.44755 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:10:25.784 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:25 smithi097 ceph-mon[64853]: from='client.54473 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:10:25.784 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:25 smithi097 ceph-mon[64853]: pgmap v312: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:10:25.784 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:25 smithi097 ceph-mon[64853]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:10:25.785 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:25 smithi097 ceph-mon[67407]: from='client.54473 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:10:25.785 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:25 smithi097 ceph-mon[67407]: pgmap v312: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:10:25.785 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:25 smithi097 ceph-mon[67407]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:10:25.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:25 smithi114 ceph-mon[58780]: from='client.54473 -' entity='client.iscsi.foo.smithi097.xndkxy' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-12T22:10:25.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:25 smithi114 ceph-mon[58780]: pgmap v312: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:10:25.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:25 smithi114 ceph-mon[58780]: from='mgr.34104 172.21.15.97:0/537871851' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-12T22:10:27.461 INFO:teuthology.orchestra.run.smithi097.stdout: "v16.2.2", 2024-04-12T22:10:27.461 INFO:teuthology.orchestra.run.smithi097.stdout: "v16.2.2-20210505", 2024-04-12T22:10:27.768 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:27 smithi097 ceph-mon[64853]: pgmap v313: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:10:27.768 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:27 smithi097 ceph-mon[64853]: from='client.34830 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:10:27.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:27 smithi097 ceph-mon[67407]: pgmap v313: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:10:27.769 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:27 smithi097 ceph-mon[67407]: from='client.34830 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:10:27.992 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:27 smithi114 ceph-mon[58780]: pgmap v313: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-12T22:10:27.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:27 smithi114 ceph-mon[58780]: from='client.34830 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T22:10:28.127 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-12T22:10:28.136 INFO:tasks.cephadm:Teardown begin 2024-04-12T22:10:28.136 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-12T22:10:28.167 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-12T22:10:28.203 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-12T22:10:28.203 DEBUG:teuthology.orchestra.run.smithi097:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-12T22:10:28.222 DEBUG:teuthology.orchestra.run.smithi114:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-12T22:10:28.260 INFO:tasks.cephadm:Stopping all daemons... 2024-04-12T22:10:28.261 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-12T22:10:28.261 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ceph-80bf7680-f916-11ee-bc8d-c7b262605968@mon.a 2024-04-12T22:10:28.410 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:28 smithi097 systemd[1]: Stopping Ceph mon.a for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:10:28.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:28 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mon-a[64849]: 2024-04-12T22:10:28.621+0000 7f07d0d81700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-12T22:10:28.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:28 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mon-a[64849]: 2024-04-12T22:10:28.621+0000 7f07d0d81700 -1 mon.a@0(leader) e4 *** Got Signal Terminated *** 2024-04-12T22:10:28.910 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:28 smithi097 podman[90797]: 2024-04-12 22:10:28.673306102 +0000 UTC m=+0.224224038 container died 3cfa16d3a85ce6f8f5277bd983073020a2cfac36820302600bf776922f7ee50a (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b277feb, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=) 2024-04-12T22:10:28.911 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:10:28 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: [12/Apr/2024:22:10:28] ENGINE Bus STOPPING 2024-04-12T22:10:28.911 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:10:28 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: [12/Apr/2024:22:10:28] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-12T22:10:28.911 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:10:28 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: [12/Apr/2024:22:10:28] ENGINE Bus STOPPED 2024-04-12T22:10:28.911 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:10:28 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: [12/Apr/2024:22:10:28] ENGINE Bus STARTING 2024-04-12T22:10:28.911 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:10:28 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: CherryPy Checker: 2024-04-12T22:10:28.911 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:10:28 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: The Application mounted at '' has an empty config. 2024-04-12T22:10:28.911 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:10:28 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:10:28.911 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:10:28 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: [12/Apr/2024:22:10:28] ENGINE Serving on http://:::9283 2024-04-12T22:10:28.911 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:10:28 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: [12/Apr/2024:22:10:28] ENGINE Bus STARTED 2024-04-12T22:10:29.408 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:10:29 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: ::ffff:172.21.15.114 - - [12/Apr/2024:22:10:29] "GET /metrics HTTP/1.1" 200 38772 "" "Prometheus/2.33.4" 2024-04-12T22:10:29.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:29 smithi097 podman[90797]: 2024-04-12 22:10:29.172929334 +0000 UTC m=+0.723847310 container cleanup 3cfa16d3a85ce6f8f5277bd983073020a2cfac36820302600bf776922f7ee50a (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, RELEASE=squid-b277feb, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD) 2024-04-12T22:10:29.408 INFO:journalctl@ceph.mon.a.smithi097.stdout:Apr 12 22:10:29 smithi097 bash[90797]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mon-a 2024-04-12T22:10:29.544 DEBUG:teuthology.orchestra.run.smithi097:> sudo pkill -f 'journalctl -f -n 0 -u ceph-80bf7680-f916-11ee-bc8d-c7b262605968@mon.a.service' 2024-04-12T22:10:29.592 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-12T22:10:29.593 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-12T22:10:29.593 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-04-12T22:10:29.593 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ceph-80bf7680-f916-11ee-bc8d-c7b262605968@mon.c 2024-04-12T22:10:29.659 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:29 smithi097 systemd[1]: Stopping Ceph mon.c for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:10:30.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:29 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mon-c[67403]: 2024-04-12T22:10:29.893+0000 7f090f3d3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-12T22:10:30.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:29 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mon-c[67403]: 2024-04-12T22:10:29.894+0000 7f090f3d3700 -1 mon.c@1(peon) e4 *** Got Signal Terminated *** 2024-04-12T22:10:30.160 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:29 smithi097 podman[91028]: 2024-04-12 22:10:29.915773599 +0000 UTC m=+0.142182254 container died 69507ba88603eec12ab2a5dadab336402e9cce7c6298eafd6aa1cd1b2988c12e (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mon-c, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-b277feb, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-12T22:10:30.579 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:30 smithi097 podman[91028]: 2024-04-12 22:10:30.303019073 +0000 UTC m=+0.529427715 container cleanup 69507ba88603eec12ab2a5dadab336402e9cce7c6298eafd6aa1cd1b2988c12e (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mon-c, GIT_CLEAN=True, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-b277feb, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-12T22:10:30.579 INFO:journalctl@ceph.mon.c.smithi097.stdout:Apr 12 22:10:30 smithi097 bash[91028]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mon-c 2024-04-12T22:10:30.717 DEBUG:teuthology.orchestra.run.smithi097:> sudo pkill -f 'journalctl -f -n 0 -u ceph-80bf7680-f916-11ee-bc8d-c7b262605968@mon.c.service' 2024-04-12T22:10:30.755 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-12T22:10:30.755 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-04-12T22:10:30.755 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-04-12T22:10:30.755 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-80bf7680-f916-11ee-bc8d-c7b262605968@mon.b 2024-04-12T22:10:31.068 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:30 smithi114 systemd[1]: Stopping Ceph mon.b for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:10:31.410 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:31 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mon-b[58775]: 2024-04-12T22:10:31.066+0000 7fc0a27d2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-12T22:10:31.410 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:31 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mon-b[58775]: 2024-04-12T22:10:31.066+0000 7fc0a27d2700 -1 mon.b@2(peon) e4 *** Got Signal Terminated *** 2024-04-12T22:10:31.410 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:31 smithi114 podman[77015]: 2024-04-12 22:10:31.118732622 +0000 UTC m=+0.167052260 container died 71a2eff159a33616171e9ceaec3ab3a21f9f32db7fe3fb91524b7097f74f0d55 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mon-b, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, RELEASE=squid-b277feb, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-12T22:10:31.742 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:31 smithi114 podman[77015]: 2024-04-12 22:10:31.601160728 +0000 UTC m=+0.649480366 container cleanup 71a2eff159a33616171e9ceaec3ab3a21f9f32db7fe3fb91524b7097f74f0d55 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-b277feb, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-12T22:10:31.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Apr 12 22:10:31 smithi114 bash[77015]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mon-b 2024-04-12T22:10:31.971 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-80bf7680-f916-11ee-bc8d-c7b262605968@mon.b.service' 2024-04-12T22:10:32.021 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-12T22:10:32.021 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-04-12T22:10:32.021 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-12T22:10:32.021 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ceph-80bf7680-f916-11ee-bc8d-c7b262605968@mgr.y 2024-04-12T22:10:32.324 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:10:32 smithi097 systemd[1]: Stopping Ceph mgr.y for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:10:32.660 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:10:32 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:10:32.323+0000 7fe44e2c3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-12T22:10:32.660 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:10:32 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y[60554]: 2024-04-12T22:10:32.323+0000 7fe44e2c3700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-04-12T22:10:32.660 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:10:32 smithi097 podman[91149]: 2024-04-12 22:10:32.354860157 +0000 UTC m=+0.138973228 container died 488170dac11fe698b3992142aa635667fb319f2377e61c1ec5d628c645d20daa (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=squid-b277feb, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-12T22:10:33.093 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:10:32 smithi097 podman[91149]: 2024-04-12 22:10:32.883710748 +0000 UTC m=+0.667823814 container cleanup 488170dac11fe698b3992142aa635667fb319f2377e61c1ec5d628c645d20daa (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-b277feb, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-12T22:10:33.093 INFO:journalctl@ceph.mgr.y.smithi097.stdout:Apr 12 22:10:32 smithi097 bash[91149]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-y 2024-04-12T22:10:33.298 DEBUG:teuthology.orchestra.run.smithi097:> sudo pkill -f 'journalctl -f -n 0 -u ceph-80bf7680-f916-11ee-bc8d-c7b262605968@mgr.y.service' 2024-04-12T22:10:33.335 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-12T22:10:33.336 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-12T22:10:33.336 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-12T22:10:33.336 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-80bf7680-f916-11ee-bc8d-c7b262605968@mgr.x 2024-04-12T22:10:33.653 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:10:33 smithi114 systemd[1]: Stopping Ceph mgr.x for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:10:33.992 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:10:33 smithi114 podman[77138]: 2024-04-12 22:10:33.653475649 +0000 UTC m=+0.125279420 container died b8f8df3bf9280c4d8c3b4c7c82fab2d551196d40bb88b4d57ecb9a66961097bc (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-b277feb, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-12T22:10:34.399 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:10:34 smithi114 podman[77138]: 2024-04-12 22:10:34.109895127 +0000 UTC m=+0.581698936 container cleanup b8f8df3bf9280c4d8c3b4c7c82fab2d551196d40bb88b4d57ecb9a66961097bc (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-b277feb, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-12T22:10:34.399 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 12 22:10:34 smithi114 bash[77138]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968-mgr-x 2024-04-12T22:10:34.526 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-80bf7680-f916-11ee-bc8d-c7b262605968@mgr.x.service' 2024-04-12T22:10:34.562 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-12T22:10:34.562 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-12T22:10:34.562 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-12T22:10:34.562 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ceph-80bf7680-f916-11ee-bc8d-c7b262605968@osd.0 2024-04-12T22:10:34.871 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:10:34 smithi097 systemd[1]: Stopping Ceph osd.0 for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:10:35.160 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:10:34 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-0[70772]: 2024-04-12T22:10:34.871+0000 7fd9be1f5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-12T22:10:35.160 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:10:34 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-0[70772]: 2024-04-12T22:10:34.871+0000 7fd9be1f5700 -1 osd.0 159 *** Got signal Terminated *** 2024-04-12T22:10:35.160 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:10:34 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-0[70772]: 2024-04-12T22:10:34.871+0000 7fd9be1f5700 -1 osd.0 159 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-12T22:10:40.660 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:10:40 smithi097 podman[91270]: 2024-04-12 22:10:40.403735045 +0000 UTC m=+5.653201979 container died 9d05fea4bf37616fb1adc9115b72fb0b082b25b09f686f5d1b5293fdfb078675 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-0, RELEASE=squid-b277feb, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-12T22:10:41.031 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:10:40 smithi097 podman[91270]: 2024-04-12 22:10:40.813211865 +0000 UTC m=+6.062678785 container cleanup 9d05fea4bf37616fb1adc9115b72fb0b082b25b09f686f5d1b5293fdfb078675 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, RELEASE=squid-b277feb, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-12T22:10:41.031 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:10:40 smithi097 bash[91270]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-0 2024-04-12T22:10:41.355 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:10:41 smithi097 podman[91288]: 2024-04-12 22:10:41.031515781 +0000 UTC m=+0.627012472 container remove 9d05fea4bf37616fb1adc9115b72fb0b082b25b09f686f5d1b5293fdfb078675 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-0, RELEASE=squid-b277feb, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0) 2024-04-12T22:10:41.660 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:10:41 smithi097 podman[91383]: 2024-04-12 22:10:41.258296348 +0000 UTC m=+0.018618211 image pull 668895944f0ec64b67a5b7564961d2756f2966e5d4b9338516300dbec9984951 quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a 2024-04-12T22:10:41.660 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:10:41 smithi097 podman[91383]: 2024-04-12 22:10:41.373994876 +0000 UTC m=+0.134316717 container create 1636f02c035d55c44afd30d5a26326dfccabcdd31b3190d425eb9e66f5067c1f (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-b277feb, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-12T22:10:41.660 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:10:41 smithi097 podman[91383]: 2024-04-12 22:10:41.602069549 +0000 UTC m=+0.362391399 container init 1636f02c035d55c44afd30d5a26326dfccabcdd31b3190d425eb9e66f5067c1f (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-b277feb) 2024-04-12T22:10:41.660 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:10:41 smithi097 podman[91383]: 2024-04-12 22:10:41.605328731 +0000 UTC m=+0.365650579 container start 1636f02c035d55c44afd30d5a26326dfccabcdd31b3190d425eb9e66f5067c1f (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-0-deactivate, RELEASE=squid-b277feb, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-12T22:10:42.160 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:10:41 smithi097 podman[91383]: 2024-04-12 22:10:41.690702299 +0000 UTC m=+0.451024142 container attach 1636f02c035d55c44afd30d5a26326dfccabcdd31b3190d425eb9e66f5067c1f (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-0-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-b277feb, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, io.buildah.version=1.33.5) 2024-04-12T22:10:42.160 INFO:journalctl@ceph.osd.0.smithi097.stdout:Apr 12 22:10:41 smithi097 podman[91383]: 2024-04-12 22:10:41.893375565 +0000 UTC m=+0.653697415 container died 1636f02c035d55c44afd30d5a26326dfccabcdd31b3190d425eb9e66f5067c1f (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-b277feb, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, io.buildah.version=1.33.5) 2024-04-12T22:10:42.673 DEBUG:teuthology.orchestra.run.smithi097:> sudo pkill -f 'journalctl -f -n 0 -u ceph-80bf7680-f916-11ee-bc8d-c7b262605968@osd.0.service' 2024-04-12T22:10:42.715 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-12T22:10:42.715 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-12T22:10:42.715 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-12T22:10:42.715 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ceph-80bf7680-f916-11ee-bc8d-c7b262605968@osd.1 2024-04-12T22:10:42.868 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:10:42 smithi097 systemd[1]: Stopping Ceph osd.1 for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:10:43.159 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:10:43 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-1[73093]: 2024-04-12T22:10:43.075+0000 7febd2aa3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-12T22:10:43.160 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:10:43 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-1[73093]: 2024-04-12T22:10:43.075+0000 7febd2aa3700 -1 osd.1 159 *** Got signal Terminated *** 2024-04-12T22:10:43.160 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:10:43 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-1[73093]: 2024-04-12T22:10:43.075+0000 7febd2aa3700 -1 osd.1 159 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-12T22:10:44.660 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:10:44 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[59315]: level=warn ts=2024-04-12T22:10:44.264Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi097.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.97:8443: connect: connection refused" 2024-04-12T22:10:44.660 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:10:44 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[59315]: level=warn ts=2024-04-12T22:10:44.264Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi114.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.114:8443: connect: connection refused" 2024-04-12T22:10:48.660 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:10:48 smithi097 podman[91496]: 2024-04-12 22:10:48.401578323 +0000 UTC m=+5.455755913 container died a9a527642889e19d8f67c54d289d38bd5e436000c65d53291c8354c2eeb12c0a (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-1, GIT_BRANCH=HEAD, RELEASE=squid-b277feb, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True) 2024-04-12T22:10:49.060 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:10:48 smithi097 podman[91496]: 2024-04-12 22:10:48.767687629 +0000 UTC m=+5.821865215 container cleanup a9a527642889e19d8f67c54d289d38bd5e436000c65d53291c8354c2eeb12c0a (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-1, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-b277feb, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-12T22:10:49.060 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:10:48 smithi097 bash[91496]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-1 2024-04-12T22:10:49.377 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:10:49 smithi097 podman[91515]: 2024-04-12 22:10:49.060680828 +0000 UTC m=+0.655564453 container remove a9a527642889e19d8f67c54d289d38bd5e436000c65d53291c8354c2eeb12c0a (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-1, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-b277feb, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-12T22:10:49.630 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:10:49 smithi097 podman[91608]: 2024-04-12 22:10:49.280636074 +0000 UTC m=+0.017947884 image pull 668895944f0ec64b67a5b7564961d2756f2966e5d4b9338516300dbec9984951 quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a 2024-04-12T22:10:49.631 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:10:49 smithi097 podman[91608]: 2024-04-12 22:10:49.436577218 +0000 UTC m=+0.173889012 container create 7847add67bab100f8d858301f484b7e3f9c4f6acc1af3870577e8ba4ded83eb0 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, RELEASE=squid-b277feb, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-12T22:10:49.631 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:10:49 smithi097 podman[91608]: 2024-04-12 22:10:49.631094712 +0000 UTC m=+0.368406505 container init 7847add67bab100f8d858301f484b7e3f9c4f6acc1af3870577e8ba4ded83eb0 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-1-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, RELEASE=squid-b277feb, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-12T22:10:49.910 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:10:49 smithi097 podman[91608]: 2024-04-12 22:10:49.634570282 +0000 UTC m=+0.371882083 container start 7847add67bab100f8d858301f484b7e3f9c4f6acc1af3870577e8ba4ded83eb0 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, RELEASE=squid-b277feb, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-12T22:10:49.910 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:10:49 smithi097 podman[91608]: 2024-04-12 22:10:49.722763385 +0000 UTC m=+0.460075185 container attach 7847add67bab100f8d858301f484b7e3f9c4f6acc1af3870577e8ba4ded83eb0 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-1-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, RELEASE=squid-b277feb, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-12T22:10:50.217 INFO:journalctl@ceph.osd.1.smithi097.stdout:Apr 12 22:10:49 smithi097 podman[91608]: 2024-04-12 22:10:49.925541527 +0000 UTC m=+0.662853327 container died 7847add67bab100f8d858301f484b7e3f9c4f6acc1af3870577e8ba4ded83eb0 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-b277feb, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-12T22:10:50.500 DEBUG:teuthology.orchestra.run.smithi097:> sudo pkill -f 'journalctl -f -n 0 -u ceph-80bf7680-f916-11ee-bc8d-c7b262605968@osd.1.service' 2024-04-12T22:10:50.543 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-12T22:10:50.543 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-12T22:10:50.544 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-12T22:10:50.544 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ceph-80bf7680-f916-11ee-bc8d-c7b262605968@osd.2 2024-04-12T22:10:50.879 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:10:50 smithi097 systemd[1]: Stopping Ceph osd.2 for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:10:51.160 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:10:50 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-2[75679]: 2024-04-12T22:10:50.878+0000 7f4344343700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-12T22:10:51.160 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:10:50 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-2[75679]: 2024-04-12T22:10:50.878+0000 7f4344343700 -1 osd.2 159 *** Got signal Terminated *** 2024-04-12T22:10:51.160 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:10:50 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-2[75679]: 2024-04-12T22:10:50.878+0000 7f4344343700 -1 osd.2 159 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-12T22:10:54.660 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:10:54 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[59315]: level=error ts=2024-04-12T22:10:54.264Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi114.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.114:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi097.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.97:8443: connect: connection refused" 2024-04-12T22:10:54.660 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:10:54 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[59315]: level=warn ts=2024-04-12T22:10:54.265Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi097.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.97:8443: connect: connection refused" 2024-04-12T22:10:54.661 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:10:54 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[59315]: level=warn ts=2024-04-12T22:10:54.265Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi114.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.114:8443: connect: connection refused" 2024-04-12T22:10:56.530 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:10:56 smithi097 podman[91712]: 2024-04-12 22:10:56.211612786 +0000 UTC m=+5.444593235 container died 2e2297afff5034813210f76ca45cd09c69b11925772c6e66624fe3c01c77bcd3 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-b277feb, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-12T22:10:56.847 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:10:56 smithi097 podman[91712]: 2024-04-12 22:10:56.612375175 +0000 UTC m=+5.845355629 container cleanup 2e2297afff5034813210f76ca45cd09c69b11925772c6e66624fe3c01c77bcd3 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-b277feb, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-12T22:10:56.847 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:10:56 smithi097 bash[91712]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-2 2024-04-12T22:10:57.159 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:10:56 smithi097 podman[91727]: 2024-04-12 22:10:56.847198973 +0000 UTC m=+0.634544078 container remove 2e2297afff5034813210f76ca45cd09c69b11925772c6e66624fe3c01c77bcd3 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-b277feb) 2024-04-12T22:10:57.483 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:10:57 smithi097 podman[91821]: 2024-04-12 22:10:57.073316008 +0000 UTC m=+0.017818766 image pull 668895944f0ec64b67a5b7564961d2756f2966e5d4b9338516300dbec9984951 quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a 2024-04-12T22:10:57.484 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:10:57 smithi097 podman[91821]: 2024-04-12 22:10:57.173097946 +0000 UTC m=+0.117600680 container create 873a7c55395c2dab1d804cf0f696793f7fd2690e19d1621c22b040c6aec809bd (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-b277feb, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-12T22:10:57.484 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:10:57 smithi097 podman[91821]: 2024-04-12 22:10:57.400547741 +0000 UTC m=+0.345050475 container init 873a7c55395c2dab1d804cf0f696793f7fd2690e19d1621c22b040c6aec809bd (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-b277feb, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-12T22:10:57.484 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:10:57 smithi097 podman[91821]: 2024-04-12 22:10:57.403698141 +0000 UTC m=+0.348200880 container start 873a7c55395c2dab1d804cf0f696793f7fd2690e19d1621c22b040c6aec809bd (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-b277feb, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-12T22:10:57.484 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:10:57 smithi097 podman[91821]: 2024-04-12 22:10:57.483934604 +0000 UTC m=+0.428437357 container attach 873a7c55395c2dab1d804cf0f696793f7fd2690e19d1621c22b040c6aec809bd (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, RELEASE=squid-b277feb, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-12T22:10:57.910 INFO:journalctl@ceph.osd.2.smithi097.stdout:Apr 12 22:10:57 smithi097 podman[91821]: 2024-04-12 22:10:57.692398946 +0000 UTC m=+0.636901695 container died 873a7c55395c2dab1d804cf0f696793f7fd2690e19d1621c22b040c6aec809bd (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=squid-b277feb, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS) 2024-04-12T22:10:58.308 DEBUG:teuthology.orchestra.run.smithi097:> sudo pkill -f 'journalctl -f -n 0 -u ceph-80bf7680-f916-11ee-bc8d-c7b262605968@osd.2.service' 2024-04-12T22:10:58.350 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-12T22:10:58.350 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-12T22:10:58.350 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-12T22:10:58.350 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ceph-80bf7680-f916-11ee-bc8d-c7b262605968@osd.3 2024-04-12T22:10:58.660 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:10:58 smithi097 systemd[1]: Stopping Ceph osd.3 for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:10:59.159 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:10:58 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-3[78009]: 2024-04-12T22:10:58.698+0000 7f94ec401700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-12T22:10:59.160 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:10:58 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-3[78009]: 2024-04-12T22:10:58.698+0000 7f94ec401700 -1 osd.3 159 *** Got signal Terminated *** 2024-04-12T22:10:59.160 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:10:58 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-3[78009]: 2024-04-12T22:10:58.698+0000 7f94ec401700 -1 osd.3 159 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-12T22:11:02.243 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:11:01 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-4[62437]: 2024-04-12T22:11:01.924+0000 7f622413b700 -1 osd.4 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:36.531932+0000 front 2024-04-12T22:10:36.531887+0000 (oldest deadline 2024-04-12T22:11:01.831663+0000) 2024-04-12T22:11:02.243 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:01 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:01.764+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.320961+0000 front 2024-04-12T22:10:39.320927+0000 (oldest deadline 2024-04-12T22:11:01.020833+0000) 2024-04-12T22:11:02.904 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:11:02 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5[64723]: 2024-04-12T22:11:02.513+0000 7fa9723a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.117716+0000 front 2024-04-12T22:10:39.117676+0000 (oldest deadline 2024-04-12T22:11:02.017633+0000) 2024-04-12T22:11:02.904 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:02 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:02.715+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.320961+0000 front 2024-04-12T22:10:39.320927+0000 (oldest deadline 2024-04-12T22:11:01.020833+0000) 2024-04-12T22:11:03.242 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:11:02 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-4[62437]: 2024-04-12T22:11:02.910+0000 7f622413b700 -1 osd.4 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:36.531932+0000 front 2024-04-12T22:10:36.531887+0000 (oldest deadline 2024-04-12T22:11:01.831663+0000) 2024-04-12T22:11:03.660 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:11:03 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-3[78009]: 2024-04-12T22:11:03.262+0000 7f94e49ed700 -1 osd.3 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.076270+0000 front 2024-04-12T22:10:39.076432+0000 (oldest deadline 2024-04-12T22:11:03.176331+0000) 2024-04-12T22:11:03.742 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:11:03 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5[64723]: 2024-04-12T22:11:03.479+0000 7fa9723a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.117716+0000 front 2024-04-12T22:10:39.117676+0000 (oldest deadline 2024-04-12T22:11:02.017633+0000) 2024-04-12T22:11:04.243 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:11:03 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-4[62437]: 2024-04-12T22:11:03.928+0000 7f622413b700 -1 osd.4 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:36.531932+0000 front 2024-04-12T22:10:36.531887+0000 (oldest deadline 2024-04-12T22:11:01.831663+0000) 2024-04-12T22:11:04.243 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:03 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:03.754+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.320961+0000 front 2024-04-12T22:10:39.320927+0000 (oldest deadline 2024-04-12T22:11:01.020833+0000) 2024-04-12T22:11:04.466 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:11:04 smithi097 podman[91925]: 2024-04-12 22:11:04.189941492 +0000 UTC m=+5.614712613 container died 415b4d3d85f2589a0068603a86da54acbaf07ac976cbaabf272ce9e5157201b8 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-3, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=squid-b277feb, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-12T22:11:04.466 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:11:04 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[59315]: level=error ts=2024-04-12T22:11:04.264Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi114.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.114:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi097.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.97:8443: connect: connection refused" 2024-04-12T22:11:04.466 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:11:04 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[59315]: level=warn ts=2024-04-12T22:11:04.265Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi097.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.97:8443: connect: connection refused" 2024-04-12T22:11:04.467 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:11:04 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[59315]: level=warn ts=2024-04-12T22:11:04.266Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi114.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.114:8443: connect: connection refused" 2024-04-12T22:11:04.742 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:04 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:04.722+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.320961+0000 front 2024-04-12T22:10:39.320927+0000 (oldest deadline 2024-04-12T22:11:01.020833+0000) 2024-04-12T22:11:04.743 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:11:04 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5[64723]: 2024-04-12T22:11:04.474+0000 7fa9723a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.117716+0000 front 2024-04-12T22:10:39.117676+0000 (oldest deadline 2024-04-12T22:11:02.017633+0000) 2024-04-12T22:11:04.825 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:11:04 smithi097 podman[91925]: 2024-04-12 22:11:04.565688162 +0000 UTC m=+5.990459277 container cleanup 415b4d3d85f2589a0068603a86da54acbaf07ac976cbaabf272ce9e5157201b8 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-3, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, RELEASE=squid-b277feb, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-12T22:11:04.825 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:11:04 smithi097 bash[91925]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-3 2024-04-12T22:11:05.159 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:11:04 smithi097 podman[91942]: 2024-04-12 22:11:04.825133935 +0000 UTC m=+0.628094378 container remove 415b4d3d85f2589a0068603a86da54acbaf07ac976cbaabf272ce9e5157201b8 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-3, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, RELEASE=squid-b277feb) 2024-04-12T22:11:05.243 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:11:04 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-4[62437]: 2024-04-12T22:11:04.880+0000 7f622413b700 -1 osd.4 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:36.531932+0000 front 2024-04-12T22:10:36.531887+0000 (oldest deadline 2024-04-12T22:11:01.831663+0000) 2024-04-12T22:11:05.478 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:11:05 smithi097 podman[92033]: 2024-04-12 22:11:05.184345154 +0000 UTC m=+0.114302667 container create 2c684e0e45108d6fd161ef4e0fa45f259bdcbb4caf39698d10b28447e0c67fcc (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-3-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=squid-b277feb, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-12T22:11:05.478 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:11:05 smithi097 podman[92033]: 2024-04-12 22:11:05.087947646 +0000 UTC m=+0.017905125 image pull 668895944f0ec64b67a5b7564961d2756f2966e5d4b9338516300dbec9984951 quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a 2024-04-12T22:11:05.479 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:11:05 smithi097 podman[92033]: 2024-04-12 22:11:05.394953981 +0000 UTC m=+0.324911448 container init 2c684e0e45108d6fd161ef4e0fa45f259bdcbb4caf39698d10b28447e0c67fcc (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, RELEASE=squid-b277feb) 2024-04-12T22:11:05.479 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:11:05 smithi097 podman[92033]: 2024-04-12 22:11:05.398046343 +0000 UTC m=+0.328003811 container start 2c684e0e45108d6fd161ef4e0fa45f259bdcbb4caf39698d10b28447e0c67fcc (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b277feb, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-12T22:11:05.479 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:11:05 smithi097 podman[92033]: 2024-04-12 22:11:05.478279482 +0000 UTC m=+0.408236961 container attach 2c684e0e45108d6fd161ef4e0fa45f259bdcbb4caf39698d10b28447e0c67fcc (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b277feb, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-12T22:11:05.743 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:11:05 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5[64723]: 2024-04-12T22:11:05.506+0000 7fa9723a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.117716+0000 front 2024-04-12T22:10:39.117676+0000 (oldest deadline 2024-04-12T22:11:02.017633+0000) 2024-04-12T22:11:05.743 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:05 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:05.385+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.689594+0000 front 2024-04-12T22:10:39.689657+0000 (oldest deadline 2024-04-12T22:11:04.989516+0000) 2024-04-12T22:11:05.909 INFO:journalctl@ceph.osd.3.smithi097.stdout:Apr 12 22:11:05 smithi097 podman[92033]: 2024-04-12 22:11:05.689603332 +0000 UTC m=+0.619560800 container died 2c684e0e45108d6fd161ef4e0fa45f259bdcbb4caf39698d10b28447e0c67fcc (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b277feb, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-12T22:11:06.242 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:05 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:05.773+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.320961+0000 front 2024-04-12T22:10:39.320927+0000 (oldest deadline 2024-04-12T22:11:01.020833+0000) 2024-04-12T22:11:06.243 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:11:05 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-4[62437]: 2024-04-12T22:11:05.876+0000 7f622413b700 -1 osd.4 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:36.531932+0000 front 2024-04-12T22:10:36.531887+0000 (oldest deadline 2024-04-12T22:11:01.831663+0000) 2024-04-12T22:11:06.272 DEBUG:teuthology.orchestra.run.smithi097:> sudo pkill -f 'journalctl -f -n 0 -u ceph-80bf7680-f916-11ee-bc8d-c7b262605968@osd.3.service' 2024-04-12T22:11:06.313 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-12T22:11:06.313 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-12T22:11:06.313 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-12T22:11:06.314 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-80bf7680-f916-11ee-bc8d-c7b262605968@osd.4 2024-04-12T22:11:06.717 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:06 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:06.434+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.689594+0000 front 2024-04-12T22:10:39.689657+0000 (oldest deadline 2024-04-12T22:11:04.989516+0000) 2024-04-12T22:11:06.717 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:11:06 smithi114 systemd[1]: Stopping Ceph osd.4 for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:11:06.717 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:11:06 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-4[62437]: 2024-04-12T22:11:06.715+0000 7f6227b47700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-12T22:11:06.718 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:11:06 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5[64723]: 2024-04-12T22:11:06.513+0000 7fa9723a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.117716+0000 front 2024-04-12T22:10:39.117676+0000 (oldest deadline 2024-04-12T22:11:02.017633+0000) 2024-04-12T22:11:06.992 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:06 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:06.769+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.320961+0000 front 2024-04-12T22:10:39.320927+0000 (oldest deadline 2024-04-12T22:11:01.020833+0000) 2024-04-12T22:11:06.993 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:11:06 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-4[62437]: 2024-04-12T22:11:06.715+0000 7f6227b47700 -1 osd.4 159 *** Got signal Terminated *** 2024-04-12T22:11:06.993 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:11:06 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-4[62437]: 2024-04-12T22:11:06.715+0000 7f6227b47700 -1 osd.4 159 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-12T22:11:06.993 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:11:06 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-4[62437]: 2024-04-12T22:11:06.858+0000 7f622413b700 -1 osd.4 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:36.531932+0000 front 2024-04-12T22:10:36.531887+0000 (oldest deadline 2024-04-12T22:11:01.831663+0000) 2024-04-12T22:11:07.742 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:07 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:07.722+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.320961+0000 front 2024-04-12T22:10:39.320927+0000 (oldest deadline 2024-04-12T22:11:01.020833+0000) 2024-04-12T22:11:07.743 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:11:07 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5[64723]: 2024-04-12T22:11:07.527+0000 7fa9723a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.117716+0000 front 2024-04-12T22:10:39.117676+0000 (oldest deadline 2024-04-12T22:11:02.017633+0000) 2024-04-12T22:11:07.743 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:07 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:07.478+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.689594+0000 front 2024-04-12T22:10:39.689657+0000 (oldest deadline 2024-04-12T22:11:04.989516+0000) 2024-04-12T22:11:08.242 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:11:07 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-4[62437]: 2024-04-12T22:11:07.872+0000 7f622413b700 -1 osd.4 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:36.531932+0000 front 2024-04-12T22:10:36.531887+0000 (oldest deadline 2024-04-12T22:11:01.831663+0000) 2024-04-12T22:11:08.743 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:08.518+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.689594+0000 front 2024-04-12T22:10:39.689657+0000 (oldest deadline 2024-04-12T22:11:04.989516+0000) 2024-04-12T22:11:08.743 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:11:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5[64723]: 2024-04-12T22:11:08.489+0000 7fa9723a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.117716+0000 front 2024-04-12T22:10:39.117676+0000 (oldest deadline 2024-04-12T22:11:02.017633+0000) 2024-04-12T22:11:08.743 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:08.673+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.320961+0000 front 2024-04-12T22:10:39.320927+0000 (oldest deadline 2024-04-12T22:11:01.020833+0000) 2024-04-12T22:11:08.744 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:11:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:11:08.653Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.97:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.97:8765: connect: connection refused" 2024-04-12T22:11:08.744 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:11:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:11:08.653Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.97:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.97:8765: connect: connection refused" 2024-04-12T22:11:08.744 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:11:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:11:08.653Z caller=refresh.go:98 level=error component="discovery manager notify" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.97:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.97:8765: connect: connection refused" 2024-04-12T22:11:08.744 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:11:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:11:08.653Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.97:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.97:8765: connect: connection refused" 2024-04-12T22:11:08.744 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:11:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:11:08.655Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.97:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.97:8765: connect: connection refused" 2024-04-12T22:11:09.242 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:11:08 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-4[62437]: 2024-04-12T22:11:08.829+0000 7f622413b700 -1 osd.4 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:36.531932+0000 front 2024-04-12T22:10:36.531887+0000 (oldest deadline 2024-04-12T22:11:01.831663+0000) 2024-04-12T22:11:09.839 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:11:09 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5[64723]: 2024-04-12T22:11:09.531+0000 7fa9723a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.117716+0000 front 2024-04-12T22:10:39.117676+0000 (oldest deadline 2024-04-12T22:11:02.017633+0000) 2024-04-12T22:11:09.840 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:09 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:09.717+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.320961+0000 front 2024-04-12T22:10:39.320927+0000 (oldest deadline 2024-04-12T22:11:01.020833+0000) 2024-04-12T22:11:09.840 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:09 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:09.536+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.689594+0000 front 2024-04-12T22:10:39.689657+0000 (oldest deadline 2024-04-12T22:11:04.989516+0000) 2024-04-12T22:11:09.840 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:09 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:09.536+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:44.990031+0000 front 2024-04-12T22:10:44.990096+0000 (oldest deadline 2024-04-12T22:11:09.089929+0000) 2024-04-12T22:11:10.243 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:11:09 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-4[62437]: 2024-04-12T22:11:09.839+0000 7f622413b700 -1 osd.4 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:36.531932+0000 front 2024-04-12T22:10:36.531887+0000 (oldest deadline 2024-04-12T22:11:01.831663+0000) 2024-04-12T22:11:10.882 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:10 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:10.685+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.320961+0000 front 2024-04-12T22:10:39.320927+0000 (oldest deadline 2024-04-12T22:11:01.020833+0000) 2024-04-12T22:11:10.882 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:10 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:10.685+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:45.721948+0000 front 2024-04-12T22:10:45.721903+0000 (oldest deadline 2024-04-12T22:11:09.821750+0000) 2024-04-12T22:11:10.883 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:11:10 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5[64723]: 2024-04-12T22:11:10.558+0000 7fa9723a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.117716+0000 front 2024-04-12T22:10:39.117676+0000 (oldest deadline 2024-04-12T22:11:02.017633+0000) 2024-04-12T22:11:10.883 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:10 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:10.549+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.689594+0000 front 2024-04-12T22:10:39.689657+0000 (oldest deadline 2024-04-12T22:11:04.989516+0000) 2024-04-12T22:11:10.883 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:10 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:10.549+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:44.990031+0000 front 2024-04-12T22:10:44.990096+0000 (oldest deadline 2024-04-12T22:11:09.089929+0000) 2024-04-12T22:11:11.242 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:11:10 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-4[62437]: 2024-04-12T22:11:10.881+0000 7f622413b700 -1 osd.4 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:36.531932+0000 front 2024-04-12T22:10:36.531887+0000 (oldest deadline 2024-04-12T22:11:01.831663+0000) 2024-04-12T22:11:11.993 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:11:11 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5[64723]: 2024-04-12T22:11:11.582+0000 7fa9723a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.117716+0000 front 2024-04-12T22:10:39.117676+0000 (oldest deadline 2024-04-12T22:11:02.017633+0000) 2024-04-12T22:11:11.993 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:11 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:11.574+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.689594+0000 front 2024-04-12T22:10:39.689657+0000 (oldest deadline 2024-04-12T22:11:04.989516+0000) 2024-04-12T22:11:11.993 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:11 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:11.574+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:44.990031+0000 front 2024-04-12T22:10:44.990096+0000 (oldest deadline 2024-04-12T22:11:09.089929+0000) 2024-04-12T22:11:11.993 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:11 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:11.701+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.320961+0000 front 2024-04-12T22:10:39.320927+0000 (oldest deadline 2024-04-12T22:11:01.020833+0000) 2024-04-12T22:11:11.993 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:11 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:11.701+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:45.721948+0000 front 2024-04-12T22:10:45.721903+0000 (oldest deadline 2024-04-12T22:11:09.821750+0000) 2024-04-12T22:11:12.468 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:11:12 smithi114 podman[77276]: 2024-04-12 22:11:12.214219614 +0000 UTC m=+5.699673388 container died fcd2e39a5866e206a9536540cb168f9f44cbd284636e137f6b79e12bae0da4b6 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-4, RELEASE=squid-b277feb, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-12T22:11:12.735 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:11:12 smithi114 podman[77276]: 2024-04-12 22:11:12.658780018 +0000 UTC m=+6.144233817 container cleanup fcd2e39a5866e206a9536540cb168f9f44cbd284636e137f6b79e12bae0da4b6 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-4, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-b277feb, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-12T22:11:12.735 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:11:12 smithi114 bash[77276]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-4 2024-04-12T22:11:12.736 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:11:12 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5[64723]: 2024-04-12T22:11:12.591+0000 7fa9723a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.117716+0000 front 2024-04-12T22:10:39.117676+0000 (oldest deadline 2024-04-12T22:11:02.017633+0000) 2024-04-12T22:11:12.736 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:11:12 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5[64723]: 2024-04-12T22:11:12.591+0000 7fa9723a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:45.819840+0000 front 2024-04-12T22:10:45.819826+0000 (oldest deadline 2024-04-12T22:11:11.719647+0000) 2024-04-12T22:11:12.736 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:12 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:12.607+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.689594+0000 front 2024-04-12T22:10:39.689657+0000 (oldest deadline 2024-04-12T22:11:04.989516+0000) 2024-04-12T22:11:12.736 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:12 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:12.607+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:44.990031+0000 front 2024-04-12T22:10:44.990096+0000 (oldest deadline 2024-04-12T22:11:09.089929+0000) 2024-04-12T22:11:12.992 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:11:12 smithi114 podman[77293]: 2024-04-12 22:11:12.868537702 +0000 UTC m=+0.649613628 container remove fcd2e39a5866e206a9536540cb168f9f44cbd284636e137f6b79e12bae0da4b6 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-4, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-b277feb, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-12T22:11:12.993 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:12 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:12.734+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.320961+0000 front 2024-04-12T22:10:39.320927+0000 (oldest deadline 2024-04-12T22:11:01.020833+0000) 2024-04-12T22:11:12.993 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:12 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:12.734+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:45.721948+0000 front 2024-04-12T22:10:45.721903+0000 (oldest deadline 2024-04-12T22:11:09.821750+0000) 2024-04-12T22:11:13.407 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:11:13 smithi114 podman[77384]: 2024-04-12 22:11:13.182424085 +0000 UTC m=+0.018870512 image pull 668895944f0ec64b67a5b7564961d2756f2966e5d4b9338516300dbec9984951 quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a 2024-04-12T22:11:13.407 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:11:13 smithi114 podman[77384]: 2024-04-12 22:11:13.294331031 +0000 UTC m=+0.130777504 container create 2bc6c947225437dcec26aaa5b84cbb9a10a332c5bd43b09b57056c1af7571621 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-4-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-b277feb, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-12T22:11:13.708 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:13 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:13.652+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.689594+0000 front 2024-04-12T22:10:39.689657+0000 (oldest deadline 2024-04-12T22:11:04.989516+0000) 2024-04-12T22:11:13.709 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:13 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:13.652+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:44.990031+0000 front 2024-04-12T22:10:44.990096+0000 (oldest deadline 2024-04-12T22:11:09.089929+0000) 2024-04-12T22:11:13.709 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:11:13 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5[64723]: 2024-04-12T22:11:13.587+0000 7fa9723a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.117716+0000 front 2024-04-12T22:10:39.117676+0000 (oldest deadline 2024-04-12T22:11:02.017633+0000) 2024-04-12T22:11:13.709 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:11:13 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5[64723]: 2024-04-12T22:11:13.587+0000 7fa9723a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:45.819840+0000 front 2024-04-12T22:10:45.819826+0000 (oldest deadline 2024-04-12T22:11:11.719647+0000) 2024-04-12T22:11:13.709 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:11:13 smithi114 podman[77384]: 2024-04-12 22:11:13.513177282 +0000 UTC m=+0.349623709 container init 2bc6c947225437dcec26aaa5b84cbb9a10a332c5bd43b09b57056c1af7571621 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=squid-b277feb, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409) 2024-04-12T22:11:13.709 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:11:13 smithi114 podman[77384]: 2024-04-12 22:11:13.516479455 +0000 UTC m=+0.352925891 container start 2bc6c947225437dcec26aaa5b84cbb9a10a332c5bd43b09b57056c1af7571621 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-4-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, RELEASE=squid-b277feb, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, ceph=True) 2024-04-12T22:11:13.709 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:11:13 smithi114 podman[77384]: 2024-04-12 22:11:13.571455621 +0000 UTC m=+0.407902077 container attach 2bc6c947225437dcec26aaa5b84cbb9a10a332c5bd43b09b57056c1af7571621 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-4-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, RELEASE=squid-b277feb, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-12T22:11:13.993 INFO:journalctl@ceph.osd.4.smithi114.stdout:Apr 12 22:11:13 smithi114 podman[77384]: 2024-04-12 22:11:13.812435371 +0000 UTC m=+0.648881793 container died 2bc6c947225437dcec26aaa5b84cbb9a10a332c5bd43b09b57056c1af7571621 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=squid-b277feb, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-12T22:11:13.993 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:13 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:13.782+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.320961+0000 front 2024-04-12T22:10:39.320927+0000 (oldest deadline 2024-04-12T22:11:01.020833+0000) 2024-04-12T22:11:13.993 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:13 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:13.782+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:45.721948+0000 front 2024-04-12T22:10:45.721903+0000 (oldest deadline 2024-04-12T22:11:09.821750+0000) 2024-04-12T22:11:14.410 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-80bf7680-f916-11ee-bc8d-c7b262605968@osd.4.service' 2024-04-12T22:11:14.455 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-12T22:11:14.455 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-12T22:11:14.455 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-12T22:11:14.456 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-80bf7680-f916-11ee-bc8d-c7b262605968@osd.5 2024-04-12T22:11:14.651 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:11:14 smithi114 systemd[1]: Stopping Ceph osd.5 for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:11:14.651 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:11:14 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5[64723]: 2024-04-12T22:11:14.612+0000 7fa9723a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.117716+0000 front 2024-04-12T22:10:39.117676+0000 (oldest deadline 2024-04-12T22:11:02.017633+0000) 2024-04-12T22:11:14.651 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:11:14 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5[64723]: 2024-04-12T22:11:14.612+0000 7fa9723a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:45.819840+0000 front 2024-04-12T22:10:45.819826+0000 (oldest deadline 2024-04-12T22:11:11.719647+0000) 2024-04-12T22:11:14.660 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:11:14 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[59315]: level=error ts=2024-04-12T22:11:14.265Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi097.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.97:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi114.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.114:8443: connect: connection refused" 2024-04-12T22:11:14.660 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:11:14 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[59315]: level=warn ts=2024-04-12T22:11:14.266Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi097.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.97:8443: connect: connection refused" 2024-04-12T22:11:14.660 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:11:14 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[59315]: level=warn ts=2024-04-12T22:11:14.266Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi114.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.114:8443: connect: connection refused" 2024-04-12T22:11:14.992 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:14 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:14.694+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.689594+0000 front 2024-04-12T22:10:39.689657+0000 (oldest deadline 2024-04-12T22:11:04.989516+0000) 2024-04-12T22:11:14.993 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:14 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:14.694+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:44.990031+0000 front 2024-04-12T22:10:44.990096+0000 (oldest deadline 2024-04-12T22:11:09.089929+0000) 2024-04-12T22:11:14.993 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:14 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:14.827+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.320961+0000 front 2024-04-12T22:10:39.320927+0000 (oldest deadline 2024-04-12T22:11:01.020833+0000) 2024-04-12T22:11:14.993 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:14 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:14.827+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:45.721948+0000 front 2024-04-12T22:10:45.721903+0000 (oldest deadline 2024-04-12T22:11:09.821750+0000) 2024-04-12T22:11:14.993 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:11:14 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5[64723]: 2024-04-12T22:11:14.753+0000 7fa979dbb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-12T22:11:14.993 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:11:14 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5[64723]: 2024-04-12T22:11:14.753+0000 7fa979dbb700 -1 osd.5 159 *** Got signal Terminated *** 2024-04-12T22:11:14.993 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:11:14 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5[64723]: 2024-04-12T22:11:14.753+0000 7fa979dbb700 -1 osd.5 159 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-12T22:11:15.992 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:11:15 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5[64723]: 2024-04-12T22:11:15.585+0000 7fa9723a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.117716+0000 front 2024-04-12T22:10:39.117676+0000 (oldest deadline 2024-04-12T22:11:02.017633+0000) 2024-04-12T22:11:15.993 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:11:15 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5[64723]: 2024-04-12T22:11:15.585+0000 7fa9723a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:45.819840+0000 front 2024-04-12T22:10:45.819826+0000 (oldest deadline 2024-04-12T22:11:11.719647+0000) 2024-04-12T22:11:15.993 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:15 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:15.682+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.689594+0000 front 2024-04-12T22:10:39.689657+0000 (oldest deadline 2024-04-12T22:11:04.989516+0000) 2024-04-12T22:11:15.993 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:15 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:15.682+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:44.990031+0000 front 2024-04-12T22:10:44.990096+0000 (oldest deadline 2024-04-12T22:11:09.089929+0000) 2024-04-12T22:11:15.993 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:15 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:15.787+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.320961+0000 front 2024-04-12T22:10:39.320927+0000 (oldest deadline 2024-04-12T22:11:01.020833+0000) 2024-04-12T22:11:15.993 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:15 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:15.787+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:45.721948+0000 front 2024-04-12T22:10:45.721903+0000 (oldest deadline 2024-04-12T22:11:09.821750+0000) 2024-04-12T22:11:16.993 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:16 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:16.692+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.689594+0000 front 2024-04-12T22:10:39.689657+0000 (oldest deadline 2024-04-12T22:11:04.989516+0000) 2024-04-12T22:11:16.993 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:16 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:16.692+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:44.990031+0000 front 2024-04-12T22:10:44.990096+0000 (oldest deadline 2024-04-12T22:11:09.089929+0000) 2024-04-12T22:11:16.993 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:11:16 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5[64723]: 2024-04-12T22:11:16.588+0000 7fa9723a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.117716+0000 front 2024-04-12T22:10:39.117676+0000 (oldest deadline 2024-04-12T22:11:02.017633+0000) 2024-04-12T22:11:16.993 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:11:16 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5[64723]: 2024-04-12T22:11:16.588+0000 7fa9723a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:45.819840+0000 front 2024-04-12T22:10:45.819826+0000 (oldest deadline 2024-04-12T22:11:11.719647+0000) 2024-04-12T22:11:16.994 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:16 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:16.742+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.320961+0000 front 2024-04-12T22:10:39.320927+0000 (oldest deadline 2024-04-12T22:11:01.020833+0000) 2024-04-12T22:11:16.994 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:16 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:16.742+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:45.721948+0000 front 2024-04-12T22:10:45.721903+0000 (oldest deadline 2024-04-12T22:11:09.821750+0000) 2024-04-12T22:11:16.994 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:16 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:16.742+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6822 osd.2 since back 2024-04-12T22:10:54.522770+0000 front 2024-04-12T22:10:54.522872+0000 (oldest deadline 2024-04-12T22:11:16.222672+0000) 2024-04-12T22:11:17.992 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:17 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:17.655+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.689594+0000 front 2024-04-12T22:10:39.689657+0000 (oldest deadline 2024-04-12T22:11:04.989516+0000) 2024-04-12T22:11:17.993 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:17 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:17.655+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:44.990031+0000 front 2024-04-12T22:10:44.990096+0000 (oldest deadline 2024-04-12T22:11:09.089929+0000) 2024-04-12T22:11:17.993 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:11:17 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5[64723]: 2024-04-12T22:11:17.623+0000 7fa9723a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.117716+0000 front 2024-04-12T22:10:39.117676+0000 (oldest deadline 2024-04-12T22:11:02.017633+0000) 2024-04-12T22:11:17.993 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:11:17 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5[64723]: 2024-04-12T22:11:17.623+0000 7fa9723a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:45.819840+0000 front 2024-04-12T22:10:45.819826+0000 (oldest deadline 2024-04-12T22:11:11.719647+0000) 2024-04-12T22:11:17.993 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:11:17 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5[64723]: 2024-04-12T22:11:17.623+0000 7fa9723a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.97:6822 osd.2 since back 2024-04-12T22:10:51.720215+0000 front 2024-04-12T22:10:51.720206+0000 (oldest deadline 2024-04-12T22:11:17.620094+0000) 2024-04-12T22:11:17.993 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:17 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:17.783+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.320961+0000 front 2024-04-12T22:10:39.320927+0000 (oldest deadline 2024-04-12T22:11:01.020833+0000) 2024-04-12T22:11:17.993 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:17 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:17.783+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:45.721948+0000 front 2024-04-12T22:10:45.721903+0000 (oldest deadline 2024-04-12T22:11:09.821750+0000) 2024-04-12T22:11:17.994 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:17 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:17.783+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6822 osd.2 since back 2024-04-12T22:10:54.522770+0000 front 2024-04-12T22:10:54.522872+0000 (oldest deadline 2024-04-12T22:11:16.222672+0000) 2024-04-12T22:11:18.993 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:18 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:18.670+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.689594+0000 front 2024-04-12T22:10:39.689657+0000 (oldest deadline 2024-04-12T22:11:04.989516+0000) 2024-04-12T22:11:18.993 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:18 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:18.670+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:44.990031+0000 front 2024-04-12T22:10:44.990096+0000 (oldest deadline 2024-04-12T22:11:09.089929+0000) 2024-04-12T22:11:18.993 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:18 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:18.771+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.320961+0000 front 2024-04-12T22:10:39.320927+0000 (oldest deadline 2024-04-12T22:11:01.020833+0000) 2024-04-12T22:11:18.993 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:18 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:18.771+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:45.721948+0000 front 2024-04-12T22:10:45.721903+0000 (oldest deadline 2024-04-12T22:11:09.821750+0000) 2024-04-12T22:11:18.993 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:18 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:18.771+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6822 osd.2 since back 2024-04-12T22:10:54.522770+0000 front 2024-04-12T22:10:54.522872+0000 (oldest deadline 2024-04-12T22:11:16.222672+0000) 2024-04-12T22:11:18.994 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:11:18 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5[64723]: 2024-04-12T22:11:18.573+0000 7fa9723a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.117716+0000 front 2024-04-12T22:10:39.117676+0000 (oldest deadline 2024-04-12T22:11:02.017633+0000) 2024-04-12T22:11:18.994 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:11:18 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5[64723]: 2024-04-12T22:11:18.573+0000 7fa9723a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:45.819840+0000 front 2024-04-12T22:10:45.819826+0000 (oldest deadline 2024-04-12T22:11:11.719647+0000) 2024-04-12T22:11:18.994 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:11:18 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5[64723]: 2024-04-12T22:11:18.573+0000 7fa9723a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.97:6822 osd.2 since back 2024-04-12T22:10:51.720215+0000 front 2024-04-12T22:10:51.720206+0000 (oldest deadline 2024-04-12T22:11:17.620094+0000) 2024-04-12T22:11:19.992 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:19 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:19.701+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.689594+0000 front 2024-04-12T22:10:39.689657+0000 (oldest deadline 2024-04-12T22:11:04.989516+0000) 2024-04-12T22:11:19.993 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:19 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:19.701+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:44.990031+0000 front 2024-04-12T22:10:44.990096+0000 (oldest deadline 2024-04-12T22:11:09.089929+0000) 2024-04-12T22:11:19.993 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:11:19 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5[64723]: 2024-04-12T22:11:19.604+0000 7fa9723a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.117716+0000 front 2024-04-12T22:10:39.117676+0000 (oldest deadline 2024-04-12T22:11:02.017633+0000) 2024-04-12T22:11:19.993 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:11:19 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5[64723]: 2024-04-12T22:11:19.604+0000 7fa9723a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:45.819840+0000 front 2024-04-12T22:10:45.819826+0000 (oldest deadline 2024-04-12T22:11:11.719647+0000) 2024-04-12T22:11:19.993 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:11:19 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5[64723]: 2024-04-12T22:11:19.604+0000 7fa9723a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.97:6822 osd.2 since back 2024-04-12T22:10:51.720215+0000 front 2024-04-12T22:10:51.720206+0000 (oldest deadline 2024-04-12T22:11:17.620094+0000) 2024-04-12T22:11:19.993 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:19 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:19.789+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.320961+0000 front 2024-04-12T22:10:39.320927+0000 (oldest deadline 2024-04-12T22:11:01.020833+0000) 2024-04-12T22:11:19.993 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:19 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:19.789+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:45.721948+0000 front 2024-04-12T22:10:45.721903+0000 (oldest deadline 2024-04-12T22:11:09.821750+0000) 2024-04-12T22:11:19.994 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:19 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:19.789+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6822 osd.2 since back 2024-04-12T22:10:54.522770+0000 front 2024-04-12T22:10:54.522872+0000 (oldest deadline 2024-04-12T22:11:16.222672+0000) 2024-04-12T22:11:20.554 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:11:20 smithi114 podman[77492]: 2024-04-12 22:11:20.266729451 +0000 UTC m=+5.622520674 container died 470c0780e42894b52fc281912e97afaba929ee847bf4fceee6b0a971098806d2 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-b277feb, org.label-schema.vendor=CentOS) 2024-04-12T22:11:20.835 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:11:20 smithi114 podman[77492]: 2024-04-12 22:11:20.73658787 +0000 UTC m=+6.092379097 container cleanup 470c0780e42894b52fc281912e97afaba929ee847bf4fceee6b0a971098806d2 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-b277feb, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2) 2024-04-12T22:11:20.835 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:11:20 smithi114 bash[77492]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5 2024-04-12T22:11:20.836 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:20 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:20.734+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.689594+0000 front 2024-04-12T22:10:39.689657+0000 (oldest deadline 2024-04-12T22:11:04.989516+0000) 2024-04-12T22:11:20.836 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:20 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:20.734+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:44.990031+0000 front 2024-04-12T22:10:44.990096+0000 (oldest deadline 2024-04-12T22:11:09.089929+0000) 2024-04-12T22:11:20.836 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:20 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:20.734+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6822 osd.2 since back 2024-04-12T22:10:54.391011+0000 front 2024-04-12T22:10:54.390991+0000 (oldest deadline 2024-04-12T22:11:20.291065+0000) 2024-04-12T22:11:21.242 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:11:21 smithi114 podman[77520]: 2024-04-12 22:11:21.012753729 +0000 UTC m=+0.734223498 container remove 470c0780e42894b52fc281912e97afaba929ee847bf4fceee6b0a971098806d2 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=squid-b277feb, org.label-schema.build-date=20240409) 2024-04-12T22:11:21.243 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:20 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:20.834+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.320961+0000 front 2024-04-12T22:10:39.320927+0000 (oldest deadline 2024-04-12T22:11:01.020833+0000) 2024-04-12T22:11:21.243 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:20 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:20.834+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:45.721948+0000 front 2024-04-12T22:10:45.721903+0000 (oldest deadline 2024-04-12T22:11:09.821750+0000) 2024-04-12T22:11:21.243 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:20 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:20.834+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6822 osd.2 since back 2024-04-12T22:10:54.522770+0000 front 2024-04-12T22:10:54.522872+0000 (oldest deadline 2024-04-12T22:11:16.222672+0000) 2024-04-12T22:11:21.657 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:11:21 smithi114 podman[77613]: 2024-04-12 22:11:21.346851037 +0000 UTC m=+0.112684762 container create 1122bdfcfe084e67c023a5baab70e617237311603a1534adebb0d3f2d4e3e7fc (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5-deactivate, RELEASE=squid-b277feb, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-12T22:11:21.657 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:11:21 smithi114 podman[77613]: 2024-04-12 22:11:21.251977674 +0000 UTC m=+0.017811413 image pull 668895944f0ec64b67a5b7564961d2756f2966e5d4b9338516300dbec9984951 quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a 2024-04-12T22:11:21.657 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:11:21 smithi114 podman[77613]: 2024-04-12 22:11:21.574358272 +0000 UTC m=+0.340192002 container init 1122bdfcfe084e67c023a5baab70e617237311603a1534adebb0d3f2d4e3e7fc (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, RELEASE=squid-b277feb, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-12T22:11:21.657 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:11:21 smithi114 podman[77613]: 2024-04-12 22:11:21.57738874 +0000 UTC m=+0.343222469 container start 1122bdfcfe084e67c023a5baab70e617237311603a1534adebb0d3f2d4e3e7fc (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-b277feb, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-12T22:11:21.657 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:11:21 smithi114 podman[77613]: 2024-04-12 22:11:21.657579501 +0000 UTC m=+0.423413236 container attach 1122bdfcfe084e67c023a5baab70e617237311603a1534adebb0d3f2d4e3e7fc (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, ceph=True, RELEASE=squid-b277feb, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-04-12T22:11:21.993 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 12 22:11:21 smithi114 podman[77613]: 2024-04-12 22:11:21.869929261 +0000 UTC m=+0.635762987 container died 1122bdfcfe084e67c023a5baab70e617237311603a1534adebb0d3f2d4e3e7fc (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-5-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=squid-b277feb, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-12T22:11:21.993 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:21.845+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.320961+0000 front 2024-04-12T22:10:39.320927+0000 (oldest deadline 2024-04-12T22:11:01.020833+0000) 2024-04-12T22:11:21.993 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:21.845+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:45.721948+0000 front 2024-04-12T22:10:45.721903+0000 (oldest deadline 2024-04-12T22:11:09.821750+0000) 2024-04-12T22:11:21.993 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:21.845+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6822 osd.2 since back 2024-04-12T22:10:54.522770+0000 front 2024-04-12T22:10:54.522872+0000 (oldest deadline 2024-04-12T22:11:16.222672+0000) 2024-04-12T22:11:21.994 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:21.731+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.689594+0000 front 2024-04-12T22:10:39.689657+0000 (oldest deadline 2024-04-12T22:11:04.989516+0000) 2024-04-12T22:11:21.994 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:21.731+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:44.990031+0000 front 2024-04-12T22:10:44.990096+0000 (oldest deadline 2024-04-12T22:11:09.089929+0000) 2024-04-12T22:11:21.994 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:21 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:21.731+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6822 osd.2 since back 2024-04-12T22:10:54.391011+0000 front 2024-04-12T22:10:54.390991+0000 (oldest deadline 2024-04-12T22:11:20.291065+0000) 2024-04-12T22:11:22.473 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-80bf7680-f916-11ee-bc8d-c7b262605968@osd.5.service' 2024-04-12T22:11:22.519 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-12T22:11:22.519 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-12T22:11:22.519 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-12T22:11:22.519 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-80bf7680-f916-11ee-bc8d-c7b262605968@osd.6 2024-04-12T22:11:22.742 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:22 smithi114 systemd[1]: Stopping Ceph osd.6 for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:11:23.243 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:22 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:22.779+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.689594+0000 front 2024-04-12T22:10:39.689657+0000 (oldest deadline 2024-04-12T22:11:04.989516+0000) 2024-04-12T22:11:23.243 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:22 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:22.779+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:44.990031+0000 front 2024-04-12T22:10:44.990096+0000 (oldest deadline 2024-04-12T22:11:09.089929+0000) 2024-04-12T22:11:23.243 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:22 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:22.779+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6822 osd.2 since back 2024-04-12T22:10:54.391011+0000 front 2024-04-12T22:10:54.390991+0000 (oldest deadline 2024-04-12T22:11:20.291065+0000) 2024-04-12T22:11:23.243 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:22 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:22.830+0000 7fc7aa25e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-12T22:11:23.243 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:22 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:22.830+0000 7fc7aa25e700 -1 osd.6 159 *** Got signal Terminated *** 2024-04-12T22:11:23.243 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:22 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:22.830+0000 7fc7aa25e700 -1 osd.6 159 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-12T22:11:23.243 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:22 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:22.823+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.320961+0000 front 2024-04-12T22:10:39.320927+0000 (oldest deadline 2024-04-12T22:11:01.020833+0000) 2024-04-12T22:11:23.244 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:22 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:22.823+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:45.721948+0000 front 2024-04-12T22:10:45.721903+0000 (oldest deadline 2024-04-12T22:11:09.821750+0000) 2024-04-12T22:11:23.244 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:22 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:22.823+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6822 osd.2 since back 2024-04-12T22:10:54.522770+0000 front 2024-04-12T22:10:54.522872+0000 (oldest deadline 2024-04-12T22:11:16.222672+0000) 2024-04-12T22:11:23.993 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:23 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:23.734+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.689594+0000 front 2024-04-12T22:10:39.689657+0000 (oldest deadline 2024-04-12T22:11:04.989516+0000) 2024-04-12T22:11:23.993 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:23 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:23.734+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:44.990031+0000 front 2024-04-12T22:10:44.990096+0000 (oldest deadline 2024-04-12T22:11:09.089929+0000) 2024-04-12T22:11:23.993 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:23 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:23.734+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6822 osd.2 since back 2024-04-12T22:10:54.391011+0000 front 2024-04-12T22:10:54.390991+0000 (oldest deadline 2024-04-12T22:11:20.291065+0000) 2024-04-12T22:11:23.993 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:23 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:23.844+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.320961+0000 front 2024-04-12T22:10:39.320927+0000 (oldest deadline 2024-04-12T22:11:01.020833+0000) 2024-04-12T22:11:23.993 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:23 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:23.844+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:45.721948+0000 front 2024-04-12T22:10:45.721903+0000 (oldest deadline 2024-04-12T22:11:09.821750+0000) 2024-04-12T22:11:23.993 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:23 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:23.844+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6822 osd.2 since back 2024-04-12T22:10:54.522770+0000 front 2024-04-12T22:10:54.522872+0000 (oldest deadline 2024-04-12T22:11:16.222672+0000) 2024-04-12T22:11:24.660 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:11:24 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[59315]: level=error ts=2024-04-12T22:11:24.266Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi097.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.97:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi114.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.114:8443: connect: connection refused" 2024-04-12T22:11:24.660 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:11:24 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[59315]: level=warn ts=2024-04-12T22:11:24.267Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi097.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.97:8443: connect: connection refused" 2024-04-12T22:11:24.660 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:11:24 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[59315]: level=warn ts=2024-04-12T22:11:24.267Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi114.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.114:8443: connect: connection refused" 2024-04-12T22:11:25.066 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:24.891+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.320961+0000 front 2024-04-12T22:10:39.320927+0000 (oldest deadline 2024-04-12T22:11:01.020833+0000) 2024-04-12T22:11:25.067 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:24.891+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:45.721948+0000 front 2024-04-12T22:10:45.721903+0000 (oldest deadline 2024-04-12T22:11:09.821750+0000) 2024-04-12T22:11:25.067 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:24.891+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6822 osd.2 since back 2024-04-12T22:10:54.522770+0000 front 2024-04-12T22:10:54.522872+0000 (oldest deadline 2024-04-12T22:11:16.222672+0000) 2024-04-12T22:11:25.067 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:24.761+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.689594+0000 front 2024-04-12T22:10:39.689657+0000 (oldest deadline 2024-04-12T22:11:04.989516+0000) 2024-04-12T22:11:25.067 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:24.761+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:44.990031+0000 front 2024-04-12T22:10:44.990096+0000 (oldest deadline 2024-04-12T22:11:09.089929+0000) 2024-04-12T22:11:25.067 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:24 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:24.761+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6822 osd.2 since back 2024-04-12T22:10:54.391011+0000 front 2024-04-12T22:10:54.390991+0000 (oldest deadline 2024-04-12T22:11:20.291065+0000) 2024-04-12T22:11:25.993 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:25 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:25.849+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.320961+0000 front 2024-04-12T22:10:39.320927+0000 (oldest deadline 2024-04-12T22:11:01.020833+0000) 2024-04-12T22:11:25.993 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:25 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:25.849+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:45.721948+0000 front 2024-04-12T22:10:45.721903+0000 (oldest deadline 2024-04-12T22:11:09.821750+0000) 2024-04-12T22:11:25.993 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:25 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:25.849+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6822 osd.2 since back 2024-04-12T22:10:54.522770+0000 front 2024-04-12T22:10:54.522872+0000 (oldest deadline 2024-04-12T22:11:16.222672+0000) 2024-04-12T22:11:25.993 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:25 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:25.809+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.689594+0000 front 2024-04-12T22:10:39.689657+0000 (oldest deadline 2024-04-12T22:11:04.989516+0000) 2024-04-12T22:11:25.993 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:25 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:25.809+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:44.990031+0000 front 2024-04-12T22:10:44.990096+0000 (oldest deadline 2024-04-12T22:11:09.089929+0000) 2024-04-12T22:11:25.993 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:25 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:25.809+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6822 osd.2 since back 2024-04-12T22:10:54.391011+0000 front 2024-04-12T22:10:54.390991+0000 (oldest deadline 2024-04-12T22:11:20.291065+0000) 2024-04-12T22:11:25.993 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:25 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:25.809+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6830 osd.3 since back 2024-04-12T22:11:01.292625+0000 front 2024-04-12T22:11:01.292677+0000 (oldest deadline 2024-04-12T22:11:24.792551+0000) 2024-04-12T22:11:27.243 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:26 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:26.898+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.320961+0000 front 2024-04-12T22:10:39.320927+0000 (oldest deadline 2024-04-12T22:11:01.020833+0000) 2024-04-12T22:11:27.243 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:26 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:26.898+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:45.721948+0000 front 2024-04-12T22:10:45.721903+0000 (oldest deadline 2024-04-12T22:11:09.821750+0000) 2024-04-12T22:11:27.243 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:26 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:26.898+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6822 osd.2 since back 2024-04-12T22:10:54.522770+0000 front 2024-04-12T22:10:54.522872+0000 (oldest deadline 2024-04-12T22:11:16.222672+0000) 2024-04-12T22:11:27.244 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:26 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:26.898+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6830 osd.3 since back 2024-04-12T22:11:02.524310+0000 front 2024-04-12T22:11:02.524289+0000 (oldest deadline 2024-04-12T22:11:26.624126+0000) 2024-04-12T22:11:27.244 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:26 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:26.784+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.689594+0000 front 2024-04-12T22:10:39.689657+0000 (oldest deadline 2024-04-12T22:11:04.989516+0000) 2024-04-12T22:11:27.244 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:26 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:26.784+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:44.990031+0000 front 2024-04-12T22:10:44.990096+0000 (oldest deadline 2024-04-12T22:11:09.089929+0000) 2024-04-12T22:11:27.244 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:26 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:26.784+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6822 osd.2 since back 2024-04-12T22:10:54.391011+0000 front 2024-04-12T22:10:54.390991+0000 (oldest deadline 2024-04-12T22:11:20.291065+0000) 2024-04-12T22:11:27.245 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:26 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:26.784+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6830 osd.3 since back 2024-04-12T22:11:01.292625+0000 front 2024-04-12T22:11:01.292677+0000 (oldest deadline 2024-04-12T22:11:24.792551+0000) 2024-04-12T22:11:28.071 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:27.939+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.320961+0000 front 2024-04-12T22:10:39.320927+0000 (oldest deadline 2024-04-12T22:11:01.020833+0000) 2024-04-12T22:11:28.071 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:27.939+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:45.721948+0000 front 2024-04-12T22:10:45.721903+0000 (oldest deadline 2024-04-12T22:11:09.821750+0000) 2024-04-12T22:11:28.071 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:27.939+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6822 osd.2 since back 2024-04-12T22:10:54.522770+0000 front 2024-04-12T22:10:54.522872+0000 (oldest deadline 2024-04-12T22:11:16.222672+0000) 2024-04-12T22:11:28.071 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:27.939+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6830 osd.3 since back 2024-04-12T22:11:02.524310+0000 front 2024-04-12T22:11:02.524289+0000 (oldest deadline 2024-04-12T22:11:26.624126+0000) 2024-04-12T22:11:28.072 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:27.804+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.689594+0000 front 2024-04-12T22:10:39.689657+0000 (oldest deadline 2024-04-12T22:11:04.989516+0000) 2024-04-12T22:11:28.072 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:27.804+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:44.990031+0000 front 2024-04-12T22:10:44.990096+0000 (oldest deadline 2024-04-12T22:11:09.089929+0000) 2024-04-12T22:11:28.072 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:27.804+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6822 osd.2 since back 2024-04-12T22:10:54.391011+0000 front 2024-04-12T22:10:54.390991+0000 (oldest deadline 2024-04-12T22:11:20.291065+0000) 2024-04-12T22:11:28.072 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:27.804+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6830 osd.3 since back 2024-04-12T22:11:01.292625+0000 front 2024-04-12T22:11:01.292677+0000 (oldest deadline 2024-04-12T22:11:24.792551+0000) 2024-04-12T22:11:28.072 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: *** Caught signal (Segmentation fault) ** 2024-04-12T22:11:28.072 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: in thread 7fc791828700 thread_name:safe_timer 2024-04-12T22:11:28.072 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev) 2024-04-12T22:11:28.072 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc7b16b4d20] 2024-04-12T22:11:28.072 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2: (BlueStore::_txc_create(BlueStore::Collection*, BlueStore::OpSequencer*, std::__cxx11::list >*, boost::intrusive_ptr)+0x40a) [0x55bcb2e7adca] 2024-04-12T22:11:28.072 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 3: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x1f8) [0x55bcb2f08968] 2024-04-12T22:11:28.072 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 4: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x53) [0x55bcb2aee733] 2024-04-12T22:11:28.073 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 5: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x7b1) [0x55bcb2d79841] 2024-04-12T22:11:28.073 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 6: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x424) [0x55bcb2a6a414] 2024-04-12T22:11:28.073 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 7: (PrimaryLogPG::simple_opc_submit(std::unique_ptr >)+0x5a) [0x55bcb2a6c47a] 2024-04-12T22:11:28.073 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 8: (PrimaryLogPG::handle_watch_timeout(std::shared_ptr)+0x873) [0x55bcb2a6e3e3] 2024-04-12T22:11:28.073 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 9: (HandleWatchTimeout::complete(int)+0x11a) [0x55bcb29cf79a] 2024-04-12T22:11:28.073 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 10: (CommonSafeTimer::timer_thread()+0x155) [0x55bcb302a735] 2024-04-12T22:11:28.073 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 11: (CommonSafeTimerThread::entry()+0x11) [0x55bcb302bf61] 2024-04-12T22:11:28.073 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fc7b16aa1ca] 2024-04-12T22:11:28.073 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 13: clone() 2024-04-12T22:11:28.073 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:27.994+0000 7fc791828700 -1 *** Caught signal (Segmentation fault) ** 2024-04-12T22:11:28.073 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: in thread 7fc791828700 thread_name:safe_timer 2024-04-12T22:11:28.073 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:28.073 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev) 2024-04-12T22:11:28.074 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc7b16b4d20] 2024-04-12T22:11:28.074 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2: (BlueStore::_txc_create(BlueStore::Collection*, BlueStore::OpSequencer*, std::__cxx11::list >*, boost::intrusive_ptr)+0x40a) [0x55bcb2e7adca] 2024-04-12T22:11:28.074 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 3: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x1f8) [0x55bcb2f08968] 2024-04-12T22:11:28.074 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 4: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x53) [0x55bcb2aee733] 2024-04-12T22:11:28.074 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 5: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x7b1) [0x55bcb2d79841] 2024-04-12T22:11:28.074 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 6: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x424) [0x55bcb2a6a414] 2024-04-12T22:11:28.074 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 7: (PrimaryLogPG::simple_opc_submit(std::unique_ptr >)+0x5a) [0x55bcb2a6c47a] 2024-04-12T22:11:28.074 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 8: (PrimaryLogPG::handle_watch_timeout(std::shared_ptr)+0x873) [0x55bcb2a6e3e3] 2024-04-12T22:11:28.074 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 9: (HandleWatchTimeout::complete(int)+0x11a) [0x55bcb29cf79a] 2024-04-12T22:11:28.074 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 10: (CommonSafeTimer::timer_thread()+0x155) [0x55bcb302a735] 2024-04-12T22:11:28.074 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 11: (CommonSafeTimerThread::entry()+0x11) [0x55bcb302bf61] 2024-04-12T22:11:28.074 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fc7b16aa1ca] 2024-04-12T22:11:28.075 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 13: clone() 2024-04-12T22:11:28.075 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T22:11:28.075 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:27 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:28.075 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: -4337> 2024-04-12T22:11:05.385+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.689594+0000 front 2024-04-12T22:10:39.689657+0000 (oldest deadline 2024-04-12T22:11:04.989516+0000) 2024-04-12T22:11:28.075 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: -3750> 2024-04-12T22:11:06.434+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.689594+0000 front 2024-04-12T22:10:39.689657+0000 (oldest deadline 2024-04-12T22:11:04.989516+0000) 2024-04-12T22:11:28.075 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: -3605> 2024-04-12T22:11:07.478+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.689594+0000 front 2024-04-12T22:10:39.689657+0000 (oldest deadline 2024-04-12T22:11:04.989516+0000) 2024-04-12T22:11:28.075 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: -3528> 2024-04-12T22:11:08.518+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.689594+0000 front 2024-04-12T22:10:39.689657+0000 (oldest deadline 2024-04-12T22:11:04.989516+0000) 2024-04-12T22:11:28.075 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: -3444> 2024-04-12T22:11:09.536+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.689594+0000 front 2024-04-12T22:10:39.689657+0000 (oldest deadline 2024-04-12T22:11:04.989516+0000) 2024-04-12T22:11:28.075 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: -3443> 2024-04-12T22:11:09.536+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:44.990031+0000 front 2024-04-12T22:10:44.990096+0000 (oldest deadline 2024-04-12T22:11:09.089929+0000) 2024-04-12T22:11:28.075 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: -2872> 2024-04-12T22:11:10.549+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.689594+0000 front 2024-04-12T22:10:39.689657+0000 (oldest deadline 2024-04-12T22:11:04.989516+0000) 2024-04-12T22:11:28.075 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: -2871> 2024-04-12T22:11:10.549+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:44.990031+0000 front 2024-04-12T22:10:44.990096+0000 (oldest deadline 2024-04-12T22:11:09.089929+0000) 2024-04-12T22:11:28.075 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: -2825> 2024-04-12T22:11:11.574+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.689594+0000 front 2024-04-12T22:10:39.689657+0000 (oldest deadline 2024-04-12T22:11:04.989516+0000) 2024-04-12T22:11:28.076 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: -2824> 2024-04-12T22:11:11.574+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:44.990031+0000 front 2024-04-12T22:10:44.990096+0000 (oldest deadline 2024-04-12T22:11:09.089929+0000) 2024-04-12T22:11:28.076 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: -2703> 2024-04-12T22:11:12.607+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.689594+0000 front 2024-04-12T22:10:39.689657+0000 (oldest deadline 2024-04-12T22:11:04.989516+0000) 2024-04-12T22:11:28.076 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: -2702> 2024-04-12T22:11:12.607+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:44.990031+0000 front 2024-04-12T22:10:44.990096+0000 (oldest deadline 2024-04-12T22:11:09.089929+0000) 2024-04-12T22:11:28.077 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: -2573> 2024-04-12T22:11:13.652+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.689594+0000 front 2024-04-12T22:10:39.689657+0000 (oldest deadline 2024-04-12T22:11:04.989516+0000) 2024-04-12T22:11:28.077 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: -2572> 2024-04-12T22:11:13.652+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:44.990031+0000 front 2024-04-12T22:10:44.990096+0000 (oldest deadline 2024-04-12T22:11:09.089929+0000) 2024-04-12T22:11:28.077 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: -2181> 2024-04-12T22:11:14.694+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.689594+0000 front 2024-04-12T22:10:39.689657+0000 (oldest deadline 2024-04-12T22:11:04.989516+0000) 2024-04-12T22:11:28.077 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: -2180> 2024-04-12T22:11:14.694+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:44.990031+0000 front 2024-04-12T22:10:44.990096+0000 (oldest deadline 2024-04-12T22:11:09.089929+0000) 2024-04-12T22:11:28.077 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: -2145> 2024-04-12T22:11:15.682+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.689594+0000 front 2024-04-12T22:10:39.689657+0000 (oldest deadline 2024-04-12T22:11:04.989516+0000) 2024-04-12T22:11:28.077 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: -2144> 2024-04-12T22:11:15.682+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:44.990031+0000 front 2024-04-12T22:10:44.990096+0000 (oldest deadline 2024-04-12T22:11:09.089929+0000) 2024-04-12T22:11:28.077 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: -2100> 2024-04-12T22:11:16.692+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.689594+0000 front 2024-04-12T22:10:39.689657+0000 (oldest deadline 2024-04-12T22:11:04.989516+0000) 2024-04-12T22:11:28.077 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: -2099> 2024-04-12T22:11:16.692+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:44.990031+0000 front 2024-04-12T22:10:44.990096+0000 (oldest deadline 2024-04-12T22:11:09.089929+0000) 2024-04-12T22:11:28.077 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: -1632> 2024-04-12T22:11:17.655+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.689594+0000 front 2024-04-12T22:10:39.689657+0000 (oldest deadline 2024-04-12T22:11:04.989516+0000) 2024-04-12T22:11:28.077 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: -1631> 2024-04-12T22:11:17.655+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:44.990031+0000 front 2024-04-12T22:10:44.990096+0000 (oldest deadline 2024-04-12T22:11:09.089929+0000) 2024-04-12T22:11:28.078 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: -1551> 2024-04-12T22:11:18.670+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.689594+0000 front 2024-04-12T22:10:39.689657+0000 (oldest deadline 2024-04-12T22:11:04.989516+0000) 2024-04-12T22:11:28.078 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: -1550> 2024-04-12T22:11:18.670+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:44.990031+0000 front 2024-04-12T22:10:44.990096+0000 (oldest deadline 2024-04-12T22:11:09.089929+0000) 2024-04-12T22:11:28.078 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: -1410> 2024-04-12T22:11:19.701+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.689594+0000 front 2024-04-12T22:10:39.689657+0000 (oldest deadline 2024-04-12T22:11:04.989516+0000) 2024-04-12T22:11:28.078 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: -1409> 2024-04-12T22:11:19.701+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:44.990031+0000 front 2024-04-12T22:10:44.990096+0000 (oldest deadline 2024-04-12T22:11:09.089929+0000) 2024-04-12T22:11:28.078 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: -1339> 2024-04-12T22:11:20.734+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.689594+0000 front 2024-04-12T22:10:39.689657+0000 (oldest deadline 2024-04-12T22:11:04.989516+0000) 2024-04-12T22:11:28.078 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: -1338> 2024-04-12T22:11:20.734+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:44.990031+0000 front 2024-04-12T22:10:44.990096+0000 (oldest deadline 2024-04-12T22:11:09.089929+0000) 2024-04-12T22:11:28.078 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: -1337> 2024-04-12T22:11:20.734+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6822 osd.2 since back 2024-04-12T22:10:54.391011+0000 front 2024-04-12T22:10:54.390991+0000 (oldest deadline 2024-04-12T22:11:20.291065+0000) 2024-04-12T22:11:28.078 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: -1269> 2024-04-12T22:11:21.731+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.689594+0000 front 2024-04-12T22:10:39.689657+0000 (oldest deadline 2024-04-12T22:11:04.989516+0000) 2024-04-12T22:11:28.078 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: -1268> 2024-04-12T22:11:21.731+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:44.990031+0000 front 2024-04-12T22:10:44.990096+0000 (oldest deadline 2024-04-12T22:11:09.089929+0000) 2024-04-12T22:11:28.078 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: -1267> 2024-04-12T22:11:21.731+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6822 osd.2 since back 2024-04-12T22:10:54.391011+0000 front 2024-04-12T22:10:54.390991+0000 (oldest deadline 2024-04-12T22:11:20.291065+0000) 2024-04-12T22:11:28.078 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: -1042> 2024-04-12T22:11:22.779+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.689594+0000 front 2024-04-12T22:10:39.689657+0000 (oldest deadline 2024-04-12T22:11:04.989516+0000) 2024-04-12T22:11:28.078 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: -1041> 2024-04-12T22:11:22.779+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:44.990031+0000 front 2024-04-12T22:10:44.990096+0000 (oldest deadline 2024-04-12T22:11:09.089929+0000) 2024-04-12T22:11:28.079 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: -1040> 2024-04-12T22:11:22.779+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6822 osd.2 since back 2024-04-12T22:10:54.391011+0000 front 2024-04-12T22:10:54.390991+0000 (oldest deadline 2024-04-12T22:11:20.291065+0000) 2024-04-12T22:11:28.079 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: -1015> 2024-04-12T22:11:22.830+0000 7fc7aa25e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-12T22:11:28.079 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: -1014> 2024-04-12T22:11:22.830+0000 7fc7aa25e700 -1 osd.6 159 *** Got signal Terminated *** 2024-04-12T22:11:28.079 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: -1012> 2024-04-12T22:11:22.830+0000 7fc7aa25e700 -1 osd.6 159 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-12T22:11:28.079 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: -897> 2024-04-12T22:11:23.734+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.689594+0000 front 2024-04-12T22:10:39.689657+0000 (oldest deadline 2024-04-12T22:11:04.989516+0000) 2024-04-12T22:11:28.079 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: -896> 2024-04-12T22:11:23.734+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:44.990031+0000 front 2024-04-12T22:10:44.990096+0000 (oldest deadline 2024-04-12T22:11:09.089929+0000) 2024-04-12T22:11:28.079 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: -895> 2024-04-12T22:11:23.734+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6822 osd.2 since back 2024-04-12T22:10:54.391011+0000 front 2024-04-12T22:10:54.390991+0000 (oldest deadline 2024-04-12T22:11:20.291065+0000) 2024-04-12T22:11:28.079 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: -764> 2024-04-12T22:11:24.761+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.689594+0000 front 2024-04-12T22:10:39.689657+0000 (oldest deadline 2024-04-12T22:11:04.989516+0000) 2024-04-12T22:11:28.079 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: -763> 2024-04-12T22:11:24.761+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:44.990031+0000 front 2024-04-12T22:10:44.990096+0000 (oldest deadline 2024-04-12T22:11:09.089929+0000) 2024-04-12T22:11:28.079 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: -762> 2024-04-12T22:11:24.761+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6822 osd.2 since back 2024-04-12T22:10:54.391011+0000 front 2024-04-12T22:10:54.390991+0000 (oldest deadline 2024-04-12T22:11:20.291065+0000) 2024-04-12T22:11:28.079 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: -355> 2024-04-12T22:11:25.809+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.689594+0000 front 2024-04-12T22:10:39.689657+0000 (oldest deadline 2024-04-12T22:11:04.989516+0000) 2024-04-12T22:11:28.079 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: -354> 2024-04-12T22:11:25.809+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:44.990031+0000 front 2024-04-12T22:10:44.990096+0000 (oldest deadline 2024-04-12T22:11:09.089929+0000) 2024-04-12T22:11:28.080 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: -353> 2024-04-12T22:11:25.809+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6822 osd.2 since back 2024-04-12T22:10:54.391011+0000 front 2024-04-12T22:10:54.390991+0000 (oldest deadline 2024-04-12T22:11:20.291065+0000) 2024-04-12T22:11:28.080 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: -352> 2024-04-12T22:11:25.809+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6830 osd.3 since back 2024-04-12T22:11:01.292625+0000 front 2024-04-12T22:11:01.292677+0000 (oldest deadline 2024-04-12T22:11:24.792551+0000) 2024-04-12T22:11:28.080 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: -232> 2024-04-12T22:11:26.784+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.689594+0000 front 2024-04-12T22:10:39.689657+0000 (oldest deadline 2024-04-12T22:11:04.989516+0000) 2024-04-12T22:11:28.080 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: -231> 2024-04-12T22:11:26.784+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:44.990031+0000 front 2024-04-12T22:10:44.990096+0000 (oldest deadline 2024-04-12T22:11:09.089929+0000) 2024-04-12T22:11:28.080 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: -230> 2024-04-12T22:11:26.784+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6822 osd.2 since back 2024-04-12T22:10:54.391011+0000 front 2024-04-12T22:10:54.390991+0000 (oldest deadline 2024-04-12T22:11:20.291065+0000) 2024-04-12T22:11:28.080 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: -229> 2024-04-12T22:11:26.784+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6830 osd.3 since back 2024-04-12T22:11:01.292625+0000 front 2024-04-12T22:11:01.292677+0000 (oldest deadline 2024-04-12T22:11:24.792551+0000) 2024-04-12T22:11:28.080 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: -122> 2024-04-12T22:11:27.804+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.689594+0000 front 2024-04-12T22:10:39.689657+0000 (oldest deadline 2024-04-12T22:11:04.989516+0000) 2024-04-12T22:11:28.080 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: -121> 2024-04-12T22:11:27.804+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:44.990031+0000 front 2024-04-12T22:10:44.990096+0000 (oldest deadline 2024-04-12T22:11:09.089929+0000) 2024-04-12T22:11:28.080 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: -120> 2024-04-12T22:11:27.804+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6822 osd.2 since back 2024-04-12T22:10:54.391011+0000 front 2024-04-12T22:10:54.390991+0000 (oldest deadline 2024-04-12T22:11:20.291065+0000) 2024-04-12T22:11:28.080 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: -119> 2024-04-12T22:11:27.804+0000 7fc7a284a700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.97:6830 osd.3 since back 2024-04-12T22:11:01.292625+0000 front 2024-04-12T22:11:01.292677+0000 (oldest deadline 2024-04-12T22:11:24.792551+0000) 2024-04-12T22:11:28.080 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 0> 2024-04-12T22:11:27.994+0000 7fc791828700 -1 *** Caught signal (Segmentation fault) ** 2024-04-12T22:11:28.080 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: in thread 7fc791828700 thread_name:safe_timer 2024-04-12T22:11:28.080 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:28.081 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: ceph version 19.0.0-2053-gb277feb2 (b277feb2c7beb4597d8ecc39d58d5c36b81f613a) squid (dev) 2024-04-12T22:11:28.081 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc7b16b4d20] 2024-04-12T22:11:28.081 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2: (BlueStore::_txc_create(BlueStore::Collection*, BlueStore::OpSequencer*, std::__cxx11::list >*, boost::intrusive_ptr)+0x40a) [0x55bcb2e7adca] 2024-04-12T22:11:28.081 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 3: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x1f8) [0x55bcb2f08968] 2024-04-12T22:11:28.081 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 4: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x53) [0x55bcb2aee733] 2024-04-12T22:11:28.081 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 5: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x7b1) [0x55bcb2d79841] 2024-04-12T22:11:28.081 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 6: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x424) [0x55bcb2a6a414] 2024-04-12T22:11:28.081 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 7: (PrimaryLogPG::simple_opc_submit(std::unique_ptr >)+0x5a) [0x55bcb2a6c47a] 2024-04-12T22:11:28.081 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 8: (PrimaryLogPG::handle_watch_timeout(std::shared_ptr)+0x873) [0x55bcb2a6e3e3] 2024-04-12T22:11:28.081 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 9: (HandleWatchTimeout::complete(int)+0x11a) [0x55bcb29cf79a] 2024-04-12T22:11:28.081 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 10: (CommonSafeTimer::timer_thread()+0x155) [0x55bcb302a735] 2024-04-12T22:11:28.081 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 11: (CommonSafeTimerThread::entry()+0x11) [0x55bcb302bf61] 2024-04-12T22:11:28.082 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fc7b16aa1ca] 2024-04-12T22:11:28.082 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 13: clone() 2024-04-12T22:11:28.082 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T22:11:28.082 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6[67023]: 2024-04-12T22:11:28.082 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 podman[77719]: 2024-04-12 22:11:28.070871136 +0000 UTC m=+5.370766210 container died 6b80aaeca08bb3819b8a392239756b62b64b2f5a3e753af4da236521d31ee897 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-b277feb, ceph=True, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-12T22:11:28.715 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 podman[77719]: 2024-04-12 22:11:28.514426006 +0000 UTC m=+5.814321128 container cleanup 6b80aaeca08bb3819b8a392239756b62b64b2f5a3e753af4da236521d31ee897 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-b277feb) 2024-04-12T22:11:28.715 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 bash[77719]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6 2024-04-12T22:11:28.992 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 podman[77734]: 2024-04-12 22:11:28.715594496 +0000 UTC m=+0.642192972 container remove 6b80aaeca08bb3819b8a392239756b62b64b2f5a3e753af4da236521d31ee897 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6, ceph=True, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-b277feb, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-12T22:11:28.993 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:28 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@osd.6.service: Main process exited, code=exited, status=139/n/a 2024-04-12T22:11:28.993 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:28.894+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.320961+0000 front 2024-04-12T22:10:39.320927+0000 (oldest deadline 2024-04-12T22:11:01.020833+0000) 2024-04-12T22:11:28.993 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:28.894+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:45.721948+0000 front 2024-04-12T22:10:45.721903+0000 (oldest deadline 2024-04-12T22:11:09.821750+0000) 2024-04-12T22:11:28.993 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:28.894+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6822 osd.2 since back 2024-04-12T22:10:54.522770+0000 front 2024-04-12T22:10:54.522872+0000 (oldest deadline 2024-04-12T22:11:16.222672+0000) 2024-04-12T22:11:28.993 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:28 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:28.894+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6830 osd.3 since back 2024-04-12T22:11:02.524310+0000 front 2024-04-12T22:11:02.524289+0000 (oldest deadline 2024-04-12T22:11:26.624126+0000) 2024-04-12T22:11:29.435 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:29 smithi114 podman[77831]: 2024-04-12 22:11:29.032932189 +0000 UTC m=+0.017451542 image pull 668895944f0ec64b67a5b7564961d2756f2966e5d4b9338516300dbec9984951 quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a 2024-04-12T22:11:29.435 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:29 smithi114 podman[77831]: 2024-04-12 22:11:29.158013618 +0000 UTC m=+0.142532997 container create 0ba9f99687bb016c01a2031344b68cc66a04a37b961ca905696ea1ed63b05948 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-b277feb, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True) 2024-04-12T22:11:29.435 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:29 smithi114 podman[77831]: 2024-04-12 22:11:29.351817845 +0000 UTC m=+0.336337182 container init 0ba9f99687bb016c01a2031344b68cc66a04a37b961ca905696ea1ed63b05948 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6-deactivate, RELEASE=squid-b277feb, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-12T22:11:29.435 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:29 smithi114 podman[77831]: 2024-04-12 22:11:29.355010071 +0000 UTC m=+0.339529408 container start 0ba9f99687bb016c01a2031344b68cc66a04a37b961ca905696ea1ed63b05948 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, RELEASE=squid-b277feb, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-12T22:11:29.435 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:29 smithi114 podman[77831]: 2024-04-12 22:11:29.435028399 +0000 UTC m=+0.419547752 container attach 0ba9f99687bb016c01a2031344b68cc66a04a37b961ca905696ea1ed63b05948 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6-deactivate, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=squid-b277feb, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS) 2024-04-12T22:11:29.743 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 12 22:11:29 smithi114 podman[77831]: 2024-04-12 22:11:29.664341114 +0000 UTC m=+0.648860452 container died 0ba9f99687bb016c01a2031344b68cc66a04a37b961ca905696ea1ed63b05948 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-6-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-b277feb) 2024-04-12T22:11:30.204 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:29 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:29.890+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.320961+0000 front 2024-04-12T22:10:39.320927+0000 (oldest deadline 2024-04-12T22:11:01.020833+0000) 2024-04-12T22:11:30.204 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:29 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:29.890+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:45.721948+0000 front 2024-04-12T22:10:45.721903+0000 (oldest deadline 2024-04-12T22:11:09.821750+0000) 2024-04-12T22:11:30.204 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:29 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:29.890+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6822 osd.2 since back 2024-04-12T22:10:54.522770+0000 front 2024-04-12T22:10:54.522872+0000 (oldest deadline 2024-04-12T22:11:16.222672+0000) 2024-04-12T22:11:30.204 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:29 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:29.890+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6830 osd.3 since back 2024-04-12T22:11:02.524310+0000 front 2024-04-12T22:11:02.524289+0000 (oldest deadline 2024-04-12T22:11:26.624126+0000) 2024-04-12T22:11:30.222 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-80bf7680-f916-11ee-bc8d-c7b262605968@osd.6.service' 2024-04-12T22:11:30.262 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-12T22:11:30.262 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-12T22:11:30.262 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-12T22:11:30.262 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-80bf7680-f916-11ee-bc8d-c7b262605968@osd.7 2024-04-12T22:11:30.492 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:30 smithi114 systemd[1]: Stopping Ceph osd.7 for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:11:30.847 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:30 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:30.582+0000 7fc430e52700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-12T22:11:30.847 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:30 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:30.582+0000 7fc430e52700 -1 osd.7 159 *** Got signal Terminated *** 2024-04-12T22:11:30.847 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:30 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:30.582+0000 7fc430e52700 -1 osd.7 159 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-12T22:11:30.847 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:30 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:30.846+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.320961+0000 front 2024-04-12T22:10:39.320927+0000 (oldest deadline 2024-04-12T22:11:01.020833+0000) 2024-04-12T22:11:30.847 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:30 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:30.846+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:45.721948+0000 front 2024-04-12T22:10:45.721903+0000 (oldest deadline 2024-04-12T22:11:09.821750+0000) 2024-04-12T22:11:30.847 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:30 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:30.846+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6822 osd.2 since back 2024-04-12T22:10:54.522770+0000 front 2024-04-12T22:10:54.522872+0000 (oldest deadline 2024-04-12T22:11:16.222672+0000) 2024-04-12T22:11:32.144 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:30 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:30.846+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6830 osd.3 since back 2024-04-12T22:11:02.524310+0000 front 2024-04-12T22:11:02.524289+0000 (oldest deadline 2024-04-12T22:11:26.624126+0000) 2024-04-12T22:11:32.144 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:31 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:31.850+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.320961+0000 front 2024-04-12T22:10:39.320927+0000 (oldest deadline 2024-04-12T22:11:01.020833+0000) 2024-04-12T22:11:32.144 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:31 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:31.850+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:45.721948+0000 front 2024-04-12T22:10:45.721903+0000 (oldest deadline 2024-04-12T22:11:09.821750+0000) 2024-04-12T22:11:32.144 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:31 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:31.850+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6822 osd.2 since back 2024-04-12T22:10:54.522770+0000 front 2024-04-12T22:10:54.522872+0000 (oldest deadline 2024-04-12T22:11:16.222672+0000) 2024-04-12T22:11:32.144 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:31 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:31.850+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6830 osd.3 since back 2024-04-12T22:11:02.524310+0000 front 2024-04-12T22:11:02.524289+0000 (oldest deadline 2024-04-12T22:11:26.624126+0000) 2024-04-12T22:11:33.242 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:32 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:32.881+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.320961+0000 front 2024-04-12T22:10:39.320927+0000 (oldest deadline 2024-04-12T22:11:01.020833+0000) 2024-04-12T22:11:33.243 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:32 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:32.881+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:45.721948+0000 front 2024-04-12T22:10:45.721903+0000 (oldest deadline 2024-04-12T22:11:09.821750+0000) 2024-04-12T22:11:33.243 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:32 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:32.881+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6822 osd.2 since back 2024-04-12T22:10:54.522770+0000 front 2024-04-12T22:10:54.522872+0000 (oldest deadline 2024-04-12T22:11:16.222672+0000) 2024-04-12T22:11:33.243 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:32 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:32.881+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6830 osd.3 since back 2024-04-12T22:11:02.524310+0000 front 2024-04-12T22:11:02.524289+0000 (oldest deadline 2024-04-12T22:11:26.624126+0000) 2024-04-12T22:11:33.243 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:32 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:32.881+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.114:6804 osd.4 since back 2024-04-12T22:11:06.624572+0000 front 2024-04-12T22:11:06.624563+0000 (oldest deadline 2024-04-12T22:11:32.524664+0000) 2024-04-12T22:11:34.242 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:33 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:33.883+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.320961+0000 front 2024-04-12T22:10:39.320927+0000 (oldest deadline 2024-04-12T22:11:01.020833+0000) 2024-04-12T22:11:34.243 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:33 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:33.883+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:45.721948+0000 front 2024-04-12T22:10:45.721903+0000 (oldest deadline 2024-04-12T22:11:09.821750+0000) 2024-04-12T22:11:34.243 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:33 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:33.883+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6822 osd.2 since back 2024-04-12T22:10:54.522770+0000 front 2024-04-12T22:10:54.522872+0000 (oldest deadline 2024-04-12T22:11:16.222672+0000) 2024-04-12T22:11:34.243 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:33 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:33.883+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6830 osd.3 since back 2024-04-12T22:11:02.524310+0000 front 2024-04-12T22:11:02.524289+0000 (oldest deadline 2024-04-12T22:11:26.624126+0000) 2024-04-12T22:11:34.243 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:33 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:33.883+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.114:6804 osd.4 since back 2024-04-12T22:11:06.624572+0000 front 2024-04-12T22:11:06.624563+0000 (oldest deadline 2024-04-12T22:11:32.524664+0000) 2024-04-12T22:11:34.660 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:11:34 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[59315]: level=error ts=2024-04-12T22:11:34.267Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi114.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.114:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi097.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.97:8443: connect: connection refused" 2024-04-12T22:11:34.660 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:11:34 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[59315]: level=warn ts=2024-04-12T22:11:34.267Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi097.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.97:8443: connect: connection refused" 2024-04-12T22:11:34.660 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:11:34 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[59315]: level=warn ts=2024-04-12T22:11:34.268Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi114.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.114:8443: connect: connection refused" 2024-04-12T22:11:35.243 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:34 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:34.876+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6806 osd.0 since back 2024-04-12T22:10:39.320961+0000 front 2024-04-12T22:10:39.320927+0000 (oldest deadline 2024-04-12T22:11:01.020833+0000) 2024-04-12T22:11:35.243 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:34 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:34.876+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6814 osd.1 since back 2024-04-12T22:10:45.721948+0000 front 2024-04-12T22:10:45.721903+0000 (oldest deadline 2024-04-12T22:11:09.821750+0000) 2024-04-12T22:11:35.243 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:34 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:34.876+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6822 osd.2 since back 2024-04-12T22:10:54.522770+0000 front 2024-04-12T22:10:54.522872+0000 (oldest deadline 2024-04-12T22:11:16.222672+0000) 2024-04-12T22:11:35.243 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:34 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:34.876+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.97:6830 osd.3 since back 2024-04-12T22:11:02.524310+0000 front 2024-04-12T22:11:02.524289+0000 (oldest deadline 2024-04-12T22:11:26.624126+0000) 2024-04-12T22:11:35.243 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:34 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7[69323]: 2024-04-12T22:11:34.876+0000 7fc42943e700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.114:6804 osd.4 since back 2024-04-12T22:11:06.624572+0000 front 2024-04-12T22:11:06.624563+0000 (oldest deadline 2024-04-12T22:11:32.524664+0000) 2024-04-12T22:11:36.175 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:35 smithi114 podman[77937]: 2024-04-12 22:11:35.909405737 +0000 UTC m=+5.431591143 container died 1c3e102eaf74e3bde0f201adfbb7f20b6c4eb14e8de83c30cd00bd066d17da36 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b277feb, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-12T22:11:36.492 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:36 smithi114 podman[77937]: 2024-04-12 22:11:36.257130912 +0000 UTC m=+5.779316315 container cleanup 1c3e102eaf74e3bde0f201adfbb7f20b6c4eb14e8de83c30cd00bd066d17da36 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-b277feb, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-12T22:11:36.493 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:36 smithi114 bash[77937]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7 2024-04-12T22:11:36.868 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:36 smithi114 podman[77967]: 2024-04-12 22:11:36.549973248 +0000 UTC m=+0.639696221 container remove 1c3e102eaf74e3bde0f201adfbb7f20b6c4eb14e8de83c30cd00bd066d17da36 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b277feb, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=) 2024-04-12T22:11:37.119 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:36 smithi114 podman[78061]: 2024-04-12 22:11:36.771308141 +0000 UTC m=+0.017288109 image pull 668895944f0ec64b67a5b7564961d2756f2966e5d4b9338516300dbec9984951 quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a 2024-04-12T22:11:37.119 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:36 smithi114 podman[78061]: 2024-04-12 22:11:36.875574181 +0000 UTC m=+0.121554150 container create b2c4e4c7b4d0a6c2632dfc554bc18fc3c507f7e99a8ed0cbc50b164e80f593c9 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7-deactivate, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-b277feb, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-12T22:11:37.420 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:37 smithi114 podman[78061]: 2024-04-12 22:11:37.119726005 +0000 UTC m=+0.365705973 container init b2c4e4c7b4d0a6c2632dfc554bc18fc3c507f7e99a8ed0cbc50b164e80f593c9 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-b277feb, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-12T22:11:37.421 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:37 smithi114 podman[78061]: 2024-04-12 22:11:37.122952065 +0000 UTC m=+0.368932036 container start b2c4e4c7b4d0a6c2632dfc554bc18fc3c507f7e99a8ed0cbc50b164e80f593c9 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, RELEASE=squid-b277feb, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=) 2024-04-12T22:11:37.421 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:37 smithi114 podman[78061]: 2024-04-12 22:11:37.20298959 +0000 UTC m=+0.448969566 container attach b2c4e4c7b4d0a6c2632dfc554bc18fc3c507f7e99a8ed0cbc50b164e80f593c9 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-b277feb, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-12T22:11:37.697 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:37 smithi114 podman[78061]: 2024-04-12 22:11:37.421211095 +0000 UTC m=+0.667191058 container died b2c4e4c7b4d0a6c2632dfc554bc18fc3c507f7e99a8ed0cbc50b164e80f593c9 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7-deactivate, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-b277feb) 2024-04-12T22:11:37.953 INFO:journalctl@ceph.osd.7.smithi114.stdout:Apr 12 22:11:37 smithi114 podman[78087]: 2024-04-12 22:11:37.939371443 +0000 UTC m=+0.512750126 container remove b2c4e4c7b4d0a6c2632dfc554bc18fc3c507f7e99a8ed0cbc50b164e80f593c9 (image=quay.ceph.io/ceph-ci/ceph:b277feb2c7beb4597d8ecc39d58d5c36b81f613a, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , RELEASE=squid-b277feb, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-12T22:11:37.958 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-80bf7680-f916-11ee-bc8d-c7b262605968@osd.7.service' 2024-04-12T22:11:37.999 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-12T22:11:37.999 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-12T22:11:38.000 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-04-12T22:11:38.000 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-80bf7680-f916-11ee-bc8d-c7b262605968@prometheus.a 2024-04-12T22:11:38.243 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:11:38 smithi114 systemd[1]: Stopping Ceph prometheus.a for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:11:38.243 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:11:38 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:11:38.191Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-04-12T22:11:38.243 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:11:38 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:11:38.191Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-04-12T22:11:38.243 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:11:38 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:11:38.191Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-04-12T22:11:38.243 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:11:38 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:11:38.191Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-04-12T22:11:38.243 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:11:38 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:11:38.191Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-04-12T22:11:38.243 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:11:38 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:11:38.191Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-04-12T22:11:38.243 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:11:38 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:11:38.191Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-04-12T22:11:38.243 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:11:38 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:11:38.191Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-04-12T22:11:38.243 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:11:38 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:11:38.191Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-04-12T22:11:38.243 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:11:38 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:11:38.191Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-04-12T22:11:38.243 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:11:38 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:11:38.191Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-04-12T22:11:38.244 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:11:38 smithi114 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a[54170]: ts=2024-04-12T22:11:38.191Z caller=main.go:1066 level=info msg="See you next time!" 2024-04-12T22:11:38.244 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:11:38 smithi114 podman[78140]: 2024-04-12 22:11:38.201441183 +0000 UTC m=+0.115625546 container died 0ad5437ed63f492c7a5f2d0da45740bd3f641bc39f6f12d09a4953f220df0618 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-12T22:11:38.742 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:11:38 smithi114 podman[78140]: 2024-04-12 22:11:38.557706426 +0000 UTC m=+0.471890789 container cleanup 0ad5437ed63f492c7a5f2d0da45740bd3f641bc39f6f12d09a4953f220df0618 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-12T22:11:38.743 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:11:38 smithi114 bash[78140]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus-a 2024-04-12T22:11:38.743 INFO:journalctl@ceph.prometheus.a.smithi114.stdout:Apr 12 22:11:38 smithi114 bash[78166]: Error: no container with name or ID "ceph-80bf7680-f916-11ee-bc8d-c7b262605968-prometheus.a" found: no such container 2024-04-12T22:11:38.830 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-80bf7680-f916-11ee-bc8d-c7b262605968@prometheus.a.service' 2024-04-12T22:11:38.868 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-12T22:11:38.868 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-04-12T22:11:38.868 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 --force --keep-logs 2024-04-12T22:11:40.660 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:11:40 smithi097 systemd[1]: Stopping Ceph node-exporter.a for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:11:40.660 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:11:40 smithi097 podman[92398]: 2024-04-12 22:11:40.493619091 +0000 UTC m=+0.122679808 container died 5edff9053b37195db0d2bbb5a5068e9d434a8737f6d3a4477c0b07df545e2423 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-12T22:11:40.994 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:11:40 smithi097 podman[92398]: 2024-04-12 22:11:40.824808483 +0000 UTC m=+0.453869206 container cleanup 5edff9053b37195db0d2bbb5a5068e9d434a8737f6d3a4477c0b07df545e2423 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-12T22:11:40.994 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:11:40 smithi097 bash[92398]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a 2024-04-12T22:11:40.994 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:11:40 smithi097 podman[92411]: 2024-04-12 22:11:40.992568572 +0000 UTC m=+0.502946174 container remove 5edff9053b37195db0d2bbb5a5068e9d434a8737f6d3a4477c0b07df545e2423 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-12T22:11:40.995 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:11:40 smithi097 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-04-12T22:11:41.345 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:11:41 smithi097 systemd[1]: Stopping Ceph alertmanager.a for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:11:41.346 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:11:41 smithi097 bash[92424]: Error: no container with name or ID "ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter.a" found: no such container 2024-04-12T22:11:41.346 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:11:41 smithi097 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-04-12T22:11:41.346 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:11:41 smithi097 systemd[1]: Stopped Ceph node-exporter.a for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:11:41.346 INFO:journalctl@ceph.node-exporter.a.smithi097.stdout:Apr 12 22:11:41 smithi097 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@node-exporter.a.service: Consumed 3.607s CPU time. 2024-04-12T22:11:41.662 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:11:41 smithi097 ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a[59315]: level=info ts=2024-04-12T22:11:41.345Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-04-12T22:11:41.662 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:11:41 smithi097 podman[92478]: 2024-04-12 22:11:41.396848752 +0000 UTC m=+0.156576765 container died 7b4c491fe0a09dfaa984d0ebaf9e8372d860e9bd97be6aec60fec459caf95983 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-12T22:11:42.047 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:11:41 smithi097 podman[92478]: 2024-04-12 22:11:41.77114634 +0000 UTC m=+0.530874346 container cleanup 7b4c491fe0a09dfaa984d0ebaf9e8372d860e9bd97be6aec60fec459caf95983 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-12T22:11:42.047 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:11:41 smithi097 bash[92478]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a 2024-04-12T22:11:42.047 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:11:41 smithi097 bash[92504]: Error: no container with name or ID "ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager.a" found: no such container 2024-04-12T22:11:42.314 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:11:42 smithi097 podman[92492]: 2024-04-12 22:11:42.047151329 +0000 UTC m=+0.643383331 container remove 7b4c491fe0a09dfaa984d0ebaf9e8372d860e9bd97be6aec60fec459caf95983 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-12T22:11:42.315 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:11:42 smithi097 podman[92492]: 2024-04-12 22:11:42.122976515 +0000 UTC m=+0.719208523 volume remove d6ee338d9e02c888fd6788570a1a335ebc03fcae2e1269ae1d1f3bc0aa266c05 2024-04-12T22:11:42.315 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:11:42 smithi097 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-04-12T22:11:42.315 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:11:42 smithi097 systemd[1]: Stopped Ceph alertmanager.a for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:11:42.315 INFO:journalctl@ceph.alertmanager.a.smithi097.stdout:Apr 12 22:11:42 smithi097 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@alertmanager.a.service: Consumed 1.035s CPU time. 2024-04-12T22:12:16.330 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 --force --keep-logs 2024-04-12T22:12:18.103 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:12:17 smithi114 systemd[1]: Stopping Ceph node-exporter.b for 80bf7680-f916-11ee-bc8d-c7b262605968... 2024-04-12T22:12:18.103 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:12:17 smithi114 podman[78510]: 2024-04-12 22:12:17.976812481 +0000 UTC m=+0.115214165 container died 904afc45dc2f5bb072538cbc23d7be546f41eaf0a22a24eda03f0250ce8ead29 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-12T22:12:18.743 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:12:18 smithi114 podman[78510]: 2024-04-12 22:12:18.399979568 +0000 UTC m=+0.538381257 container cleanup 904afc45dc2f5bb072538cbc23d7be546f41eaf0a22a24eda03f0250ce8ead29 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-12T22:12:18.743 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:12:18 smithi114 bash[78510]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b 2024-04-12T22:12:18.743 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:12:18 smithi114 bash[78538]: Error: no container with name or ID "ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter.b" found: no such container 2024-04-12T22:12:18.743 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:12:18 smithi114 podman[78524]: 2024-04-12 22:12:18.600920723 +0000 UTC m=+0.626460112 container remove 904afc45dc2f5bb072538cbc23d7be546f41eaf0a22a24eda03f0250ce8ead29 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-80bf7680-f916-11ee-bc8d-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-12T22:12:18.743 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:12:18 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-04-12T22:12:18.743 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:12:18 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-04-12T22:12:18.743 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:12:18 smithi114 systemd[1]: Stopped Ceph node-exporter.b for 80bf7680-f916-11ee-bc8d-c7b262605968. 2024-04-12T22:12:18.743 INFO:journalctl@ceph.node-exporter.b.smithi114.stdout:Apr 12 22:12:18 smithi114 systemd[1]: ceph-80bf7680-f916-11ee-bc8d-c7b262605968@node-exporter.b.service: Consumed 3.589s CPU time. 2024-04-12T22:12:41.641 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-12T22:12:41.677 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-12T22:12:41.708 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-12T22:12:41.710 DEBUG:teuthology.misc:Transferring archived files from smithi097:/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-12_21:08:02-orch-squid-distro-default-smithi/7654552/remote/smithi097/crash 2024-04-12T22:12:41.711 DEBUG:teuthology.orchestra.run.smithi097:> sudo tar c -f - -C /var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/crash -- . 2024-04-12T22:12:41.761 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/crash: Cannot open: No such file or directory 2024-04-12T22:12:41.761 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2024-04-12T22:12:41.763 DEBUG:teuthology.misc:Transferring archived files from smithi114:/var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-12_21:08:02-orch-squid-distro-default-smithi/7654552/remote/smithi114/crash 2024-04-12T22:12:41.764 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/crash -- . 2024-04-12T22:12:41.805 INFO:teuthology.orchestra.run.smithi114.stderr:tar: /var/lib/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/crash: Cannot open: No such file or directory 2024-04-12T22:12:41.806 INFO:teuthology.orchestra.run.smithi114.stderr:tar: Error is not recoverable: exiting now 2024-04-12T22:12:41.807 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-12T22:12:41.807 DEBUG:teuthology.orchestra.run.smithi097:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-04-12T22:12:41.839 INFO:teuthology.orchestra.run.smithi097.stdout:2024-04-12T21:54:02.431933+0000 mon.a (mon.0) 465 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-12T22:12:41.839 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-04-12T22:12:41.839 DEBUG:teuthology.orchestra.run.smithi097:> sudo egrep '\[SEC\]' /var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-04-12T22:12:41.910 DEBUG:teuthology.orchestra.run.smithi097:> sudo egrep '\[ERR\]' /var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-04-12T22:12:41.981 DEBUG:teuthology.orchestra.run.smithi097:> sudo egrep '\[WRN\]' /var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-04-12T22:12:42.051 INFO:teuthology.orchestra.run.smithi097.stdout:2024-04-12T21:54:02.431933+0000 mon.a (mon.0) 465 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-12T22:12:42.051 INFO:tasks.cephadm:Compressing logs... 2024-04-12T22:12:42.051 DEBUG:teuthology.orchestra.run.smithi097:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-12T22:12:42.093 DEBUG:teuthology.orchestra.run.smithi114:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-12T22:12:42.120 INFO:teuthology.orchestra.run.smithi097.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-12T22:12:42.120 INFO:teuthology.orchestra.run.smithi097.stderr:: No such file or directory 2024-04-12T22:12:42.121 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph-mon.a.log 2024-04-12T22:12:42.121 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph.audit.log 2024-04-12T22:12:42.121 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph-osd.1.log 2024-04-12T22:12:42.121 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph-mon.a.log: /var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph-osd.3.log 2024-04-12T22:12:42.121 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph-client.rgw.smpl.smithi097.exxkgl.log 2024-04-12T22:12:42.121 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph-mgr.y.log 2024-04-12T22:12:42.121 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph-client.rgw.smpl.smithi097.exxkgl.log: gzip -5 --verbose -- /var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/tcmu-runner.log 2024-04-12T22:12:42.121 INFO:teuthology.orchestra.run.smithi097.stderr: 76.8% -- replaced with /var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph-client.rgw.smpl.smithi097.exxkgl.log.gz 2024-04-12T22:12:42.122 INFO:teuthology.orchestra.run.smithi114.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-12T22:12:42.122 INFO:teuthology.orchestra.run.smithi114.stderr:: No such file or directory 2024-04-12T22:12:42.123 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph-osd.6.log 2024-04-12T22:12:42.123 INFO:teuthology.orchestra.run.smithi114.stderr:gzip/var/log/ceph/cephadm.log: -5 --verbose -- /var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph-client.rgw.foo.smithi114.ulfbyo.log 2024-04-12T22:12:42.123 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph-client.rgw.smpl.smithi114.lrgvph.log 2024-04-12T22:12:42.126 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph-client.rgw.foo.smithi114.ulfbyo.log: gzip -5 --verbose -- /var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph.audit.log 2024-04-12T22:12:42.126 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph-client.rgw.smpl.smithi114.lrgvph.log: 76.8% -- replaced with /var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph-client.rgw.foo.smithi114.ulfbyo.log.gz 2024-04-12T22:12:42.126 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph-osd.4.log 2024-04-12T22:12:42.126 INFO:teuthology.orchestra.run.smithi114.stderr: 76.3% -- replaced with /var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph-client.rgw.smpl.smithi114.lrgvph.log.gz 2024-04-12T22:12:42.126 INFO:teuthology.orchestra.run.smithi114.stderr:gzip/var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph.audit.log: -5 --verbose -- /var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph-osd.7.log 2024-04-12T22:12:42.126 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph-osd.5.log 2024-04-12T22:12:42.127 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph-mgr.x.log 2024-04-12T22:12:42.128 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph-mgr.y.log: 94.2% -- replaced with /var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph.audit.log.gz 2024-04-12T22:12:42.128 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph-osd.0.log 2024-04-12T22:12:42.128 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/tcmu-runner.log: gzip -5 --verbose -- /var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph-client.rgw.foo.smithi097.sggtkh.log 2024-04-12T22:12:42.128 INFO:teuthology.orchestra.run.smithi097.stderr: 84.5% -- replaced with /var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/tcmu-runner.log.gz 2024-04-12T22:12:42.129 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph.cephadm.log 2024-04-12T22:12:42.129 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph-client.rgw.foo.smithi097.sggtkh.log: 76.8% -- replaced with /var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph-client.rgw.foo.smithi097.sggtkh.log.gz 2024-04-12T22:12:42.129 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph.log 2024-04-12T22:12:42.130 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph-osd.5.log: 90.4% -- replaced with /var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph.audit.log.gz 2024-04-12T22:12:42.130 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph.cephadm.log 2024-04-12T22:12:42.131 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph.cephadm.log: 90.7% -- replaced with /var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph.cephadm.log.gz 2024-04-12T22:12:42.131 INFO:teuthology.orchestra.run.smithi097.stderr: 91.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-12T22:12:42.135 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph-osd.2.log 2024-04-12T22:12:42.135 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph-volume.log 2024-04-12T22:12:42.143 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph-osd.2.log: 92.9% -- replaced with /var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph.log.gz 2024-04-12T22:12:42.143 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph-mon.c.log 2024-04-12T22:12:42.144 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph-mgr.x.log: 90.9% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-12T22:12:42.144 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph.log 2024-04-12T22:12:42.144 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph-volume.log 2024-04-12T22:12:42.144 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph.cephadm.log: /var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph.log: 83.6% -- replaced with /var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph.cephadm.log.gz 2024-04-12T22:12:42.145 INFO:teuthology.orchestra.run.smithi114.stderr: 87.3% -- replaced with /var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph.log.gz 2024-04-12T22:12:42.145 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph-mon.b.log 2024-04-12T22:12:42.153 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph-volume.log: 91.3% -- replaced with /var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph-volume.log.gz 2024-04-12T22:12:42.159 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph-volume.log: /var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph-mon.b.log: 91.4% -- replaced with /var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph-volume.log.gz 2024-04-12T22:12:42.198 INFO:teuthology.orchestra.run.smithi114.stderr: 89.7% -- replaced with /var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph-mgr.x.log.gz 2024-04-12T22:12:42.474 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph-mon.c.log: 88.6% -- replaced with /var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph-mgr.y.log.gz 2024-04-12T22:12:42.546 INFO:teuthology.orchestra.run.smithi114.stderr: 92.4% -- replaced with /var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph-mon.b.log.gz 2024-04-12T22:12:42.581 INFO:teuthology.orchestra.run.smithi097.stderr: 92.2% -- replaced with /var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph-mon.c.log.gz 2024-04-12T22:12:43.208 INFO:teuthology.orchestra.run.smithi097.stderr: 93.4% -- replaced with /var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph-osd.2.log.gz 2024-04-12T22:12:43.302 INFO:teuthology.orchestra.run.smithi097.stderr: 91.0% -- replaced with /var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph-mon.a.log.gz 2024-04-12T22:12:43.319 INFO:teuthology.orchestra.run.smithi114.stderr: 93.5% -- replaced with /var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph-osd.6.log.gz 2024-04-12T22:12:43.522 INFO:teuthology.orchestra.run.smithi114.stderr: 94.0% -- replaced with /var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph-osd.7.log.gz 2024-04-12T22:12:43.529 INFO:teuthology.orchestra.run.smithi114.stderr: 93.7% -- replaced with /var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph-osd.5.log.gz 2024-04-12T22:12:43.694 INFO:teuthology.orchestra.run.smithi114.stderr: 93.8% -- replaced with /var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph-osd.4.log.gz 2024-04-12T22:12:43.696 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-04-12T22:12:43.696 INFO:teuthology.orchestra.run.smithi114.stderr:real 0m1.588s 2024-04-12T22:12:43.696 INFO:teuthology.orchestra.run.smithi114.stderr:user 0m5.842s 2024-04-12T22:12:43.696 INFO:teuthology.orchestra.run.smithi114.stderr:sys 0m0.244s 2024-04-12T22:12:43.701 INFO:teuthology.orchestra.run.smithi097.stderr: 93.7% -- replaced with /var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph-osd.0.log.gz 2024-04-12T22:12:43.798 INFO:teuthology.orchestra.run.smithi097.stderr: 93.7% -- replaced with /var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph-osd.1.log.gz 2024-04-12T22:12:43.840 INFO:teuthology.orchestra.run.smithi097.stderr: 93.7% -- replaced with /var/log/ceph/80bf7680-f916-11ee-bc8d-c7b262605968/ceph-osd.3.log.gz 2024-04-12T22:12:43.843 INFO:teuthology.orchestra.run.smithi097.stderr: 2024-04-12T22:12:43.843 INFO:teuthology.orchestra.run.smithi097.stderr:real 0m1.738s 2024-04-12T22:12:43.843 INFO:teuthology.orchestra.run.smithi097.stderr:user 0m7.785s 2024-04-12T22:12:43.843 INFO:teuthology.orchestra.run.smithi097.stderr:sys 0m0.273s 2024-04-12T22:12:43.843 INFO:tasks.cephadm:Archiving logs... 2024-04-12T22:12:43.843 DEBUG:teuthology.misc:Transferring archived files from smithi097:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-12_21:08:02-orch-squid-distro-default-smithi/7654552/remote/smithi097/log 2024-04-12T22:12:43.844 DEBUG:teuthology.orchestra.run.smithi097:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-12T22:12:44.865 DEBUG:teuthology.misc:Transferring archived files from smithi114:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-12_21:08:02-orch-squid-distro-default-smithi/7654552/remote/smithi114/log 2024-04-12T22:12:44.866 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-12T22:12:45.695 INFO:tasks.cephadm:Removing cluster... 2024-04-12T22:12:45.695 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 --force 2024-04-12T22:12:45.979 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 80bf7680-f916-11ee-bc8d-c7b262605968 --force 2024-04-12T22:12:46.266 INFO:tasks.cephadm:Removing cephadm ... 2024-04-12T22:12:46.266 DEBUG:teuthology.orchestra.run.smithi097:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-12T22:12:46.281 DEBUG:teuthology.orchestra.run.smithi114:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-12T22:12:46.295 INFO:tasks.cephadm:Teardown complete 2024-04-12T22:12:46.296 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-12T22:12:46.305 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-12T22:12:46.305 DEBUG:teuthology.orchestra.run.smithi097:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-12T22:12:46.323 DEBUG:teuthology.orchestra.run.smithi114:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-12T22:12:46.335 INFO:teuthology.orchestra.run.smithi097.stderr:bash: line 1: ntpq: command not found 2024-04-12T22:12:46.339 INFO:teuthology.orchestra.run.smithi097.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-12T22:12:46.339 INFO:teuthology.orchestra.run.smithi097.stdout:=============================================================================== 2024-04-12T22:12:46.339 INFO:teuthology.orchestra.run.smithi097.stdout:^- hv01.front.sepia.ceph.com 2 6 377 49 +1055us[+1063us] +/- 366ms 2024-04-12T22:12:46.340 INFO:teuthology.orchestra.run.smithi097.stdout:^* hv02.front.sepia.ceph.com 3 6 377 48 -455us[ -446us] +/- 60ms 2024-04-12T22:12:46.340 INFO:teuthology.orchestra.run.smithi097.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 49 +531us[ +540us] +/- 70ms 2024-04-12T22:12:46.340 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T22:12:46.349 INFO:teuthology.orchestra.run.smithi114.stderr:bash: line 1: ntpq: command not found 2024-04-12T22:12:46.354 INFO:teuthology.orchestra.run.smithi114.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-12T22:12:46.354 INFO:teuthology.orchestra.run.smithi114.stdout:=============================================================================== 2024-04-12T22:12:46.354 INFO:teuthology.orchestra.run.smithi114.stdout:^- hv01.front.sepia.ceph.com 2 7 377 111 +1093us[+1116us] +/- 365ms 2024-04-12T22:12:46.354 INFO:teuthology.orchestra.run.smithi114.stdout:^* hv02.front.sepia.ceph.com 3 6 377 47 -461us[ -437us] +/- 60ms 2024-04-12T22:12:46.354 INFO:teuthology.orchestra.run.smithi114.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 48 +538us[ +561us] +/- 70ms 2024-04-12T22:12:46.354 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T22:12:46.354 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-12T22:12:46.364 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-12T22:12:46.365 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-12T22:12:46.401 DEBUG:teuthology.orchestra.run.smithi097:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-12T22:12:46.482 DEBUG:teuthology.orchestra.run.smithi114:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-12T22:12:46.560 DEBUG:teuthology.orchestra.run.smithi097:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-12T22:12:46.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-12T22:12:46.590 DEBUG:teuthology.orchestra.run.smithi114:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-12T22:12:46.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-12T22:12:46.622 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-12T22:12:46.631 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-12T22:12:46.639 INFO:teuthology.task.internal:Duration was 1672.331964 seconds 2024-04-12T22:12:46.640 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-12T22:12:46.649 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-12T22:12:46.649 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-12T22:12:46.652 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-12T22:12:46.693 INFO:teuthology.orchestra.run.smithi097.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-12T22:12:46.706 INFO:teuthology.orchestra.run.smithi114.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-12T22:12:47.196 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-12T22:12:47.197 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi097.front.sepia.ceph.com 2024-04-12T22:12:47.197 DEBUG:teuthology.orchestra.run.smithi097:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-12T22:12:47.259 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi114.front.sepia.ceph.com 2024-04-12T22:12:47.259 DEBUG:teuthology.orchestra.run.smithi114:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-12T22:12:47.283 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-12T22:12:47.333 DEBUG:teuthology.orchestra.run.smithi097:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-12T22:12:47.335 DEBUG:teuthology.orchestra.run.smithi114:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-12T22:12:47.363 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-12T22:12:47.363 DEBUG:teuthology.orchestra.run.smithi097:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-12T22:12:47.404 DEBUG:teuthology.orchestra.run.smithi114:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-12T22:12:47.824 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-12T22:12:47.835 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-12T22:12:47.835 DEBUG:teuthology.orchestra.run.smithi097:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-12T22:12:47.862 DEBUG:teuthology.orchestra.run.smithi114:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-12T22:12:47.889 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-12T22:12:47.900 DEBUG:teuthology.orchestra.run.smithi097:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-12T22:12:47.904 DEBUG:teuthology.orchestra.run.smithi114:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-12T22:12:47.927 INFO:teuthology.orchestra.run.smithi097.stdout:kernel.core_pattern = core 2024-04-12T22:12:47.954 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern = core 2024-04-12T22:12:47.977 DEBUG:teuthology.orchestra.run.smithi097:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-12T22:12:48.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-12T22:12:48.001 DEBUG:teuthology.orchestra.run.smithi114:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-12T22:12:48.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-12T22:12:48.031 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-12T22:12:48.041 INFO:teuthology.task.internal:Transferring archived files... 2024-04-12T22:12:48.042 DEBUG:teuthology.misc:Transferring archived files from smithi097:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-12_21:08:02-orch-squid-distro-default-smithi/7654552/remote/smithi097 2024-04-12T22:12:48.043 DEBUG:teuthology.orchestra.run.smithi097:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-12T22:12:48.094 DEBUG:teuthology.misc:Transferring archived files from smithi114:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-12_21:08:02-orch-squid-distro-default-smithi/7654552/remote/smithi114 2024-04-12T22:12:48.095 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-12T22:12:48.138 INFO:teuthology.task.internal:Removing archive directory... 2024-04-12T22:12:48.138 DEBUG:teuthology.orchestra.run.smithi097:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-12T22:12:48.141 DEBUG:teuthology.orchestra.run.smithi114:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-12T22:12:48.186 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-12T22:12:48.198 INFO:teuthology.task.internal:Not uploading archives. 2024-04-12T22:12:48.198 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-12T22:12:48.208 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-12T22:12:48.208 DEBUG:teuthology.orchestra.run.smithi097:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-12T22:12:48.210 DEBUG:teuthology.orchestra.run.smithi114:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-12T22:12:48.222 INFO:teuthology.orchestra.run.smithi097.stdout: 266043 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 12 22:12 /home/ubuntu/cephtest 2024-04-12T22:12:48.240 INFO:teuthology.orchestra.run.smithi114.stdout: 265867 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 12 22:12 /home/ubuntu/cephtest 2024-04-12T22:12:48.241 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-12T22:12:48.252 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-12T22:12:48.271 INFO:teuthology.nuke:Checking targets against current locks 2024-04-12T22:12:48.299 DEBUG:teuthology.nuke:shortname: smithi097 2024-04-12T22:12:48.300 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-12T22:12:48.324 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi097.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-12_21:08:02-orch-squid-distro-default-smithi/7654552', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-12 21:36:33.259804', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG4eVILE1reKOrx59u+2caNLf58FHQ/N1J8NVOvd7RU2mNjNDFKyMY0rXLvNNV3W9KpF2Xv3F3BqBiJmBTJiWhU='} 2024-04-12T22:12:48.327 DEBUG:teuthology.nuke:shortname: smithi114 2024-04-12T22:12:48.328 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-12T22:12:48.348 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi114.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-12_21:08:02-orch-squid-distro-default-smithi/7654552', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-12 21:36:33.258584', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEGezFQLvdH5Ts9lZM2TglSlf9zAepCCLzbYdhsV+9c8NMLFhO74WYfXhVBhKrMFB3dn6rA3xKU8AZIysFjNlOQ='} 2024-04-12T22:12:48.381 INFO:teuthology.orchestra.console.smithi097:Power off 2024-04-12T22:12:48.381 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-12T22:12:48.404 DEBUG:teuthology.orchestra.console.smithi097:power off output: Chassis Power Control: Down/Off 2024-04-12T22:12:48.405 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-12T22:12:48.420 INFO:teuthology.orchestra.console.smithi114:Power off 2024-04-12T22:12:48.420 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-12T22:12:48.429 DEBUG:teuthology.orchestra.console.smithi097:check power output: Chassis Power is on 2024-04-12T22:12:48.443 DEBUG:teuthology.orchestra.console.smithi114:power off output: Chassis Power Control: Down/Off 2024-04-12T22:12:48.443 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-12T22:12:48.465 DEBUG:teuthology.orchestra.console.smithi114:check power output: Chassis Power is on 2024-04-12T22:12:52.431 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-12T22:12:52.467 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-12T22:12:52.541 DEBUG:teuthology.orchestra.console.smithi097:check power output: Chassis Power is on 2024-04-12T22:12:52.577 DEBUG:teuthology.orchestra.console.smithi114:check power output: Chassis Power is on 2024-04-12T22:12:56.546 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-12T22:12:56.578 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-12T22:12:56.657 DEBUG:teuthology.orchestra.console.smithi097:check power output: Chassis Power is on 2024-04-12T22:12:56.690 DEBUG:teuthology.orchestra.console.smithi114:check power output: Chassis Power is on 2024-04-12T22:13:00.662 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-12T22:13:00.691 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-12T22:13:00.772 DEBUG:teuthology.orchestra.console.smithi097:check power output: Chassis Power is off 2024-04-12T22:13:00.801 DEBUG:teuthology.orchestra.console.smithi114:check power output: Chassis Power is off 2024-04-12T22:13:00.873 INFO:teuthology.orchestra.console.smithi097:Power off completed 2024-04-12T22:13:00.902 INFO:teuthology.orchestra.console.smithi114:Power off completed 2024-04-12T22:13:01.006 INFO:teuthology.run:Summary data: description: orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} duration: 1672.3319635391235 failure_reason: '"2024-04-12T21:54:02.431933+0000 mon.a (mon.0) 465 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log' owner: scheduled_teuthology@teuthology success: false 2024-04-12T22:13:01.007 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-12T22:13:01.093 INFO:teuthology.run:FAIL